Authors: Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Lu Yuan, Leonid Sigal
Abstract: Recent studies customizing Multimodal Large Language Models (MLLMs) for
domain-specific tasks have yielded promising results, especially in the field
of scientific chart comprehension. These studies generally utilize visual
instruction tuning with specialized datasets to enhance question and answer
(QA) accuracy within the chart domain. However, they often neglect the
fundamental discrepancy between natural image-caption pre-training data and
digital chart image-QA data, particularly in the models’ capacity to extract
underlying numeric values from charts. This paper tackles this oversight by
exploring the training processes necessary to improve MLLMs’ comprehension of
charts. We present three key findings: (1) Incorporating raw data values in
alignment pre-training markedly improves comprehension of chart data. (2)
Replacing images with their textual representation randomly during end-to-end
fine-tuning transfer the language reasoning capability to chart interpretation
skills. (3) Requiring the model to first extract the underlying chart data and
then answer the question in the fine-tuning can further improve the accuracy.
Consequently, we introduce CHOPINLLM, an MLLM tailored for in-depth chart
comprehension. CHOPINLLM effectively interprets various types of charts,
including unannotated ones, while maintaining robust reasoning abilities.
Furthermore, we establish a new benchmark to evaluate MLLMs’ understanding of
different chart types across various comprehension levels. Experimental results
show that CHOPINLLM exhibits strong performance in understanding both annotated
and unannotated charts across a wide range of types.
Source: http://arxiv.org/abs/2407.14506v1
