Jeonghwan Kim

From perception to understanding.

Jeonghwan Kim

I’m Jeonghwan Kim, a Ph.D. candidate at UIUC’s BLENDER Lab, advised by Professor Heng Ji. I am a recipient of the Capital One Ph.D. Fellowship (2026–2027). My research is also supported by HERE Technologies. I worked with Shane Moon and Xin Luna Dong at Meta Reality Labs and Emre Eftelioglu at Amazon.

I study multimodal foundation models that perceive, ground, and reason about fine-grained spatial and temporal information. My work connects visual grounding, cross-modal alignment, and knowledge integration to make these systems more accurate and interpretable. Recently, I’ve been extending this work to visually grounded action policies for embodied AI and robotics.

I’m on the industry and academic job market for 2027. Get in touch if you would like to connect.

Research keywords: multimodal LLMs, multimodal post-training, visual tokenization & memory, vision-language-action.

Google Scholar CV Email LinkedIn X

01 / Updates

News

Featured first · scroll for more

Featured

  • Our paper Mixture of Layers: Dynamic Layer Routing for Visual Reasoning was accepted to NeurIPS 2026.

    Project

  • Our paper PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models was accepted to NeurIPS 2026 @ VLM4RWD. Oral Presentation

    Paper

  • Our paper PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding was accepted to NeurIPS 2025. Spotlight

    Project

  • Our paper ProMac: Language-Guided World Action Modeling for Single-View Egocentric Navigation is a preprint (2026).

  • I’ll be in Atlanta for NeurIPS 2026!

  • I received the Capital One Ph.D. Fellowship (2026–2027) at UIUC.

  • I joined Meta in May as a Research Scientist Intern, working on language-guided world action models for smart glasses.

  • Our paper Advancing Creative Physical Intelligence in Large Multimodal Models was accepted to NeurIPS 2026.

  • Our paper CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing was accepted to COLM 2026.

  • Our paper MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Knowledge Poisoning Attacks was accepted to ACL 2026.

  • Our paper AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints was accepted to COLM 2026.

  • Our paper Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping was accepted to ICLR 2026.

  • Our paper MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Models was accepted to EMNLP 2026 Findings.

  • Our paper PEARL: Self-Evolving Assistant for Time Management with Reinforcement Learning was accepted to ACL 2026.

    Paper

  • Our paper SYNTHIA: Novel Concept Design with Affordance Composition was accepted to ACL 2025.

  • Our paper Search and Detect: Training-Free Long Tail Object Detection via Web-Image Retrieval was accepted to CVPR 2025.

  • Our paper Aligning LLMs with Individual Preferences via Interaction was accepted to COLING 2025.

  • I joined Meta in May as a Research Scientist Intern (Part-time Student Researcher), working on pixel-level retrieval-augmented generation for multimodal models.

  • Our paper Infogent: An Agent-Based Framework for Web Information Aggregation was accepted to NAACL 2025 Findings.

    Paper

  • Our paper MIRACLE: An Online, Explainable Multimodal Interactive Concept Learning System was accepted to ACM Multimedia 2024 Demo and Video Program Track.

  • Our paper Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models was accepted to EMNLP 2024.

    Paper

  • Our paper ARMADA: Attribute-Based Multimodal Data Augmentation was accepted to WikiNLP at EMNLP 2024.

    Paper

  • I joined Amazon in May as an Applied Scientist Intern, developing a generalizable multimodal encoder for geospatial applications.

  • Our paper Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise was accepted to NAACL 2024 Findings.

    Paper

  • Our paper FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4 was accepted to EMNLP 2023 Findings.

02 / Research

Selected publications

All publications on Scholar

* indicates equal contribution.

03 / Experience

Professional Experience

Full CV

Meta

Research Scientist Intern

World-Action Models for Wearables · Redmond, WA

May 2026 – Present

Meta

Research Scientist Intern (Part-time Student Researcher)

Multimodal RAG + MLLM · Redmond, WA

May – Oct 2025

Amazon

Applied Scientist Intern

Multimodal Representation Learning · Bellevue, WA

May – Aug 2024

UIUC

Graduate Research Assistant · Ph.D.

Advisor: Heng Ji

Aug 2023 – Present

KAIST

Research Associate

IR&NLP Lab · Advisor: Sung-Hyon Myaeng

Mar 2022 – Jul 2023

KAIST

Graduate Research Assistant · M.S.

IR&NLP Lab · Advisor: Sung-Hyon Myaeng

Feb 2020 – Feb 2022

04 / Background

Education & service

UIUC

Ph.D. in Computer Science

BLENDER Lab

Present

KAIST

M.Sc. in School of Computing

IR&NLP Lab

Feb. 2022

Republic of Korea Marine Corps

Honorably Discharged

Mandatory Military Service

Dec. 2016

Handong Global University

B.Sc. in Computer Science & Electrical Engineering

Magna Cum Laude

Feb. 2020