An assistant professor in the Department of Electrical Engineering and Computer Science has been awarded the prestigious National Science Foundation Faculty Early Career Development award to support his research into the growing challenge of artificial intelligence models producing "hallucinations."

Khoa Luu's $499,728 CAREER award will allow him to continue research into the tendency of large-scale multimodal models, typically referred to as LMMs, to produce immature or inaccurate outputs known as "hallucinations." As the principal investigator, Luu will focus on improving the trustworthiness and performance of LMMs. This is most applicable in healthcare-related applications such as tobacco prevention advertising and autism behavior prediction.

Outcomes from the research will impact the field by providing a foundational and practical study needed for future research. It will also train students to conduct and use research to improve community health and mental health outcomes.

Three main factors cause hallucinations in LMMs. First, biased data distributions significantly influence LMM predictions and can produce hallucinations due to uneven data representation. Second, misalignment between input modalities may lead models to overly rely on a dominant modality, ignoring others and resulting in false predictions. Third, insufficient large-scale and diverse training data limits model knowledge, prompting hallucinations.

The goal of the project is to develop robust and trustworthy large multimodal models to reduce hallucinations in vision-language model outputs. Luu will achieve this by strengthening three areas of design.

He will first introduce new learning approaches to address hallucinations caused by imbalanced data. Next, he will design and implement novel shuffling techniques to correct misalignment across input modalities. Finally, Luu will develop adaptive learning methods to address the limitations of training data, thereby improving the performance and reliability of large multimodal models.

The research effort will lay the groundwork for developing new theoretical and practical approaches in distribution modeling, contrastive learning and shuffling techniques to address hallucinations in large multimodal models.

Luu added, "We hope the new studies in this project will help to better understand the nature of large multimodal learning problems and provide a reliability performance of these AI models. The fundamental research and delivered components in the project will help to train new AI talents and create a vehicle for the next phase in the evolution of LMM for healthcare and other stakeholders."

Luu has received seven patents and three Best Paper awards, and has co-authored over 120 papers in AI-related conferences, technical reports and journals. He currently serves as an associate editor of the multimedia tools and applications journal Springer Nature.

He also serves as the area chair in several top-tier AI conferences, including IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023-2025), Annual Conference on Neural Information Processing Systems (2024-2025), International Conference on Machine Learning 2025, International Conference on Learning Representations 2025 and Winter Conference on Applications of Computer Vision 2025.