Paper Info
Title
Psgraph: How Tencent Trains Extremely Large-Scale Graphs With Spark?
Abstract
Spark has extensively used in many applications of Tencent, due to its easy deployment, pipeline capability, and close integration with the Hadoop ecosystem. As the graph computing engine of Spark, GraphX is also widely deployed to process large-scale graph data in Tencent. However, when the size of the graph data is up to billion-scale, GraphX encounters serious performance degradation. Worse, Graphx cannot support the rising advancement of graph embedding (GE) and graph neural network (GNN) algorithms. To address these challenges, we develop a new graph processing system, called PSGraph, which uses Spark executor and PyTorch to perform calculation, and develops a distributed parameter server to store frequently accessed models. PSGraph can train extremely large-scale graph data in Tencent with the parameter server architecture, and enable the training of GE and GNN algorithms. Moreover, PSGraph still benefits from the advantages of Spark via staying inside the Spark ecosystem, and can directly replace GraphX without modification to the existing application framework. Our experiments show that PSGraph outperforms GraphX significantly.
Year
DOI
Venue
2020
10.1109/ICDE48307.2020.00137
2020 IEEE 36TH INTERNATIONAL CONFERENCE ON DATA ENGINEERING (ICDE 2020)
Keywords
DocType
ISSN
graph algorithm, Spark, parameter server
Conference
1084-4627
Citations 
PageRank 
References 
0
0.34
19
Authors
8
Name
Order
Citations
PageRank
Jiawei Jiang18914.60
Pin Xiao200.34
Lele Yu3706.93
Xiaosen Li421.50
Jiefeng Cheng500.34
Xupeng Miao6143.33
Zhipeng Zhang7112.20
Bin Cui81843124.59