Authors: Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf
Abstract: Assessing the capabilities of large language models (LLMs) is often
challenging, in part, because it is hard to find tasks to which they have not
been exposed during training. We take one step to address this challenge by
turning to a new task: focusing on symbolic graphics programs, which are a
popular representation for graphics content that procedurally generates visual
data. LLMs have shown exciting promise towards program synthesis, but do they
understand symbolic graphics programs? Unlike conventional programs, symbolic
graphics programs can be translated to graphics content. Here, we characterize
an LLM’s understanding of symbolic programs in terms of their ability to answer
questions related to the graphics content. This task is challenging as the
questions are difficult to answer from the symbolic programs alone — yet, they
would be easy to answer from the corresponding graphics content as we verify
through a human experiment. To understand symbolic programs, LLMs may need to
possess the ability to imagine how the corresponding graphics content would
look without directly accessing the rendered visual content. We use this task
to evaluate LLMs by creating a large benchmark for the semantic understanding
of symbolic graphics programs. This benchmark is built via program-graphics
correspondence, hence requiring minimal human efforts. We evaluate current LLMs
on our benchmark to elucidate a preliminary assessment of their ability to
reason about visual scenes from programs. We find that this task distinguishes
existing LLMs and models considered good at reasoning perform better. Lastly,
we introduce Symbolic Instruction Tuning (SIT) to improve this ability.
Specifically, we query GPT4-o with questions and images generated by symbolic
programs. Such data are then used to finetune an LLM. We also find that SIT
data can improve the general instruction following ability of LLMs.
Source: http://arxiv.org/abs/2408.08313v1
