Authors: Yucheng Zhou, Lingran Song, Jianbing Shen
Abstract: Existing Medical Large Vision-Language Models (Med-LVLMs), which encapsulate
extensive medical knowledge, demonstrate excellent capabilities in
understanding medical images and responding to human queries based on these
images. However, there remain challenges in visual localization in medical
images, which is crucial for abnormality detection and interpretation. To
address these issues, we propose a novel UMed-LVLM designed with Unveiling
Medical abnormalities. Specifically, we collect a Medical Abnormalities
Unveiling (MAU) dataset and propose a two-stage training method for UMed-LVLM
training. To collect MAU dataset, we propose a prompt method utilizing the
GPT-4V to generate diagnoses based on identified abnormal areas in medical
images. Moreover, the two-stage training method includes Abnormal-Aware
Instruction Tuning and Abnormal-Aware Rewarding, comprising Abnormal
Localization Rewarding and Vision Relevance Rewarding. Experimental results
demonstrate that our UMed-LVLM surpasses existing Med-LVLMs in identifying and
understanding medical abnormality. In addition, this work shows that enhancing
the abnormality detection capabilities of Med-LVLMs significantly improves
their understanding of medical images and generalization capability.
Source: http://arxiv.org/abs/2501.01377v1