Integrating Multimodal Information In Large Pretrained Transformers

ARXIV  ·  2019  ·  Impact 8

Wasifur Rahman; Md. Kamrul Hasan; Sangwu Lee; Amir Zadeh; Chengfeng Mao; Louis-Philippe Morency; Ehsan Hoque;

HIGHLIGHT

In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG).

RELATED

Papers      Patents     Grants     Venues     Experts     Code    Cite    Share    Save

ABSTRACT

Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straightforward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). Pre-trained models don't have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.

 Paraphrase     Questionfy     Wikify     Simplify     Summarize

Download