Paper Info
Title
DILAF: A framework for distributed analysis of large-scale system logs for anomaly detection.
Abstract
System logs constitute a rich source of information for detection and prediction of anomalies. However, they can include a huge volume of data, which is usually unstructured or semistructured. We introduce DILAF, a framework for distributed analysis of large-scale system logs for anomaly detection. DILAF is comprised of several processes to facilitate log parsing, feature extraction, and machine learning activities. It has two distinguishing features with respect to the existing tools. First, it does not require the availability of source code of the analyzed system. Second, it is designed to perform all the processes in a distributed manner to support scalable analysis in the context of large-scale distributed systems. We discuss the software architecture of DILAF and we introduce an implementation of it. We conducted controlled experiments based on two datasets to evaluate the effectiveness of the framework. In particular, we evaluated the performance and scalability attributes under various degrees of parallelism. Results showed that DILAF can maintain the same accuracy levels while achieving more than 30% performance improvement on average as the system scales, compared to baseline approaches that do not employ fully distributed processing.
Year
DOI
Venue
2019
10.1002/spe.2653
SOFTWARE-PRACTICE & EXPERIENCE
Keywords
DocType
Volume
anomaly detection,big data,distributed systems,log analysis,machine learning,parallel processing,software architecture
Journal
49.0
Issue
ISSN
Citations 
SP2.0
0038-0644
0
PageRank 
References 
Authors
0.34
0
3
Name
Order
Citations
PageRank
Merve Astekin142.35
Harun Zengin200.34
Hasan Sözer3101.87