Authors: Congbo Ma, Wei Emma Zhang, Dileepa Pitawela, Haojie Zhuang, Yanfeng Shu
Abstract: The utilization of Transformer-based models prospers the growth of
multi-document summarization (MDS). Given the huge impact and widespread
adoption of Transformer-based models in various natural language processing
tasks, investigating their performance and behaviors in the context of MDS
becomes crucial for advancing the field and enhancing the quality of summary.
To thoroughly examine the behaviours of Transformer-based MDS models, this
paper presents five empirical studies on (1) measuring the impact of document
boundary separators quantitatively; (2) exploring the effectiveness of
different mainstream Transformer structures; (3) examining the sensitivity of
the encoder and decoder; (4) discussing different training strategies; and (5)
discovering the repetition in a summary generation. The experimental results on
prevalent MDS datasets and eleven evaluation metrics show the influence of
document boundary separators, the granularity of different level features and
different model training strategies. The results also reveal that the decoder
exhibits greater sensitivity to noises compared to the encoder. This
underscores the important role played by the decoder, suggesting a potential
direction for future research in MDS. Furthermore, the experimental results
indicate that the repetition problem in the generated summaries has
correlations with the high uncertainty scores.
Source: http://arxiv.org/abs/2407.11948v1
