Transformer for Graphs: An Overview from Architecture Perspective

ARXIV  ·  2022  ·  Impact 5

Erxue Min; Runfa Chen; Yatao Bian; Tingyang Xu; Kangfei Zhao; Wenbing Huang; Peilin Zhao; Junzhou Huang; Sophia Ananiadou; Yu Rong;

HIGHLIGHT

In this survey, we provide a comprehensive review of various Graph Transformer models from the architectural design perspective.

RELATED

Papers      Patents     Grants     Venues     Experts     Code    Cite    Share    Save

ABSTRACT

Recently, Transformer model, which has achieved great success in many artificial intelligence fields, has demonstrated its great potential in modeling graph-structured data. Till now, a great variety of Transformers has been proposed to adapt to the graph-structured data. However, a comprehensive literature review and systematical evaluation of these Transformer variants for graphs are still unavailable. It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. In this survey, we provide a comprehensive review of various Graph Transformer models from the architectural design perspective. We first disassemble the existing models and conclude three typical ways to incorporate the graph information into the vanilla Transformer: 1) GNNs as Auxiliary Modules, 2) Improved Positional Embedding from Graphs, and 3) Improved Attention Matrix from Graphs. Furthermore, we implement the representative components in three groups and conduct a comprehensive comparison on various kinds of famous graph data benchmarks to investigate the real performance gain of each component. Our experiments confirm the benefits of current graph-specific modules on Transformer and reveal their advantages on different kinds of graph tasks.

 Paraphrase     Questionfy     Wikify     Simplify     Summarize

Download