Multimodal Paper Summarization with Hierarchical Fusion

Modern scientific articles extend beyond plain text, with publishers such as Elsevier frequently supplementing their papers with videos and graphical abstracts (infographic summaries) to enhance the reading experience. Currently, paper summarizers capable of effectively fusing two or more modalities are scarce, and the complexity of integrating these diverse modalities, often with missing or incomplete data, calls for advanced modelling technologies. This paper introduces Hier-SciSum, a new model for Multimodal Paper Summarization (MPS), Hier-SciSum incorporates a Hierarchical Multimodal Fusion (HMF) module, which effectively integrates diverse modalities by first capturing pairwise intrinsic cross-modality correlations through attention mechanisms and then refining these relationships with cross-attention masking. This hierarchical approach allows for a progressive understanding of both low-level pairwise relationships and higher-level integrated representations. Extensive experiments conducted on a newly introduced MPS dataset showcase our model's effectiveness. Impressively, Hier-SciSum generates high-quality summaries, outperforming both uni- and multi-modality models.