Paper Info
Title
Distributed Entropy Minimization Discretizer for Big Data Analysis under Apache Spark
Abstract
The astonishing rate of data generation on the Internet nowadays has caused that many classical knowledge extraction techniques have become obsolete. Data reduction techniques are required in order to reduce the complexity order held by these techniques. Among reduction techniques, discretization is one of the most important tasks in data mining process, aimed at simplifying and reducing continuous-valued data in large datasets. In spite of the great interest in this reduction mechanism, only a few simple discretization techniques have been implemented in the literature for Big Data. Thereby we propose a distributed implementation of the entropy minimization discretizer proposed by Fayyad and Irani using Apache Spark platform. Our solution goes beyond a simple parallelization, transforming the iterativity yielded by the original proposal in a single-step computation. Experimental results on two large-scale datasets show that our solution is able to improve the classification accuracy as well as boosting the underlying learning process.
Year
DOI
Venue
2015
10.1109/Trustcom-BigDataSe-ISPA.2015.559
TrustCom/BigDataSE/ISPA
Keywords
Field
DocType
distributed entropy minimization discretizer,Big Data analysis,Apache spark,Internet,knowledge extraction technique,data mining process,continuous-valued data reduction,single-step computation,classification accuracy,learning process
Discretization,Data mining,Spark (mathematics),Computer science,Boosting (machine learning),Artificial intelligence,Knowledge extraction,Big data,Test data generation,Machine learning,Data reduction,Computation
Conference
Volume
ISSN
Citations 
2
2324-9013
6
PageRank 
References 
Authors
0.44
9
4
Name
Order
Citations
PageRank
Sergio Ramírez-Gallego1986.99
Salvador García234311.50
Hector Mourino-Talin360.44
David Martínez-Rego414312.70