AIQL: Enabling efficient attack investigation from system monitoring data

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)

Abstract

The need for countering Advanced Persistent Threat (APT) attacks has led to the solutions that ubiquitously monitor system activities in each host, and perform timely attack investigation over the monitoring data for analyzing attack provenance. However, existing query systems based on relational databases and graph databases lack language constructs to express key properties of major attack behaviors, and often execute queries inefficiently since their semantics-agnostic design cannot exploit the properties of system monitoring data to speed up query execution. To address this problem, we propose a novel query system built on top of existing monitoring tools and databases, which is designed with novel types of optimizations to support timely attack investigation. Our system provides (1) domain-specific data model and storage for scaling the storage, (2) a domain-specific query language, Attack Investigation Query Language (AIQL) that integrates critical primitives for attack investigation, and (3) an optimized query engine based on the characteristics of the data and the semantics of the queries to efficiently schedule the query execution. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 857 GB of real system monitoring data (containing 2.5 billion events). Our evaluations on a real-world APT attack and a broad set of attack behaviors show that our system surpasses existing systems in both efficiency (124x over PostgreSQL, 157x over Neo4j, and 16x over Greenplum) and conciseness (SQL, Neo4j Cypher, and Splunk SPL contain at least 2.4x more constraints than AIQL).

Original languageEnglish (US)
Title of host publicationProceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018
PublisherUSENIX Association
Pages113-125
Number of pages13
ISBN (Electronic)9781939133021
StatePublished - Jan 1 2020
Event2018 USENIX Annual Technical Conference, USENIX ATC 2018 - Boston, United States
Duration: Jul 11 2018Jul 13 2018

Publication series

NameProceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018

Conference

Conference2018 USENIX Annual Technical Conference, USENIX ATC 2018
CountryUnited States
CityBoston
Period7/11/187/13/18

Fingerprint

Query languages
Monitoring
Semantics
Data structures
Engines

All Science Journal Classification (ASJC) codes

  • Computer Science(all)

Cite this

Kulkarni, S. R., & Mittal, P. (2020). AIQL: Enabling efficient attack investigation from system monitoring data. In Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018 (pp. 113-125). (Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018). USENIX Association.
Kulkarni, Sanjeev R. ; Mittal, Prateek. / AIQL : Enabling efficient attack investigation from system monitoring data. Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018. USENIX Association, 2020. pp. 113-125 (Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018).
@inproceedings{b750c5ae703d46619ee66423cb8b46ae,
title = "AIQL: Enabling efficient attack investigation from system monitoring data",
abstract = "The need for countering Advanced Persistent Threat (APT) attacks has led to the solutions that ubiquitously monitor system activities in each host, and perform timely attack investigation over the monitoring data for analyzing attack provenance. However, existing query systems based on relational databases and graph databases lack language constructs to express key properties of major attack behaviors, and often execute queries inefficiently since their semantics-agnostic design cannot exploit the properties of system monitoring data to speed up query execution. To address this problem, we propose a novel query system built on top of existing monitoring tools and databases, which is designed with novel types of optimizations to support timely attack investigation. Our system provides (1) domain-specific data model and storage for scaling the storage, (2) a domain-specific query language, Attack Investigation Query Language (AIQL) that integrates critical primitives for attack investigation, and (3) an optimized query engine based on the characteristics of the data and the semantics of the queries to efficiently schedule the query execution. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 857 GB of real system monitoring data (containing 2.5 billion events). Our evaluations on a real-world APT attack and a broad set of attack behaviors show that our system surpasses existing systems in both efficiency (124x over PostgreSQL, 157x over Neo4j, and 16x over Greenplum) and conciseness (SQL, Neo4j Cypher, and Splunk SPL contain at least 2.4x more constraints than AIQL).",
author = "Kulkarni, {Sanjeev R.} and Prateek Mittal",
year = "2020",
month = "1",
day = "1",
language = "English (US)",
series = "Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018",
publisher = "USENIX Association",
pages = "113--125",
booktitle = "Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018",

}

Kulkarni, SR & Mittal, P 2020, AIQL: Enabling efficient attack investigation from system monitoring data. in Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018. Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018, USENIX Association, pp. 113-125, 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, United States, 7/11/18.

AIQL : Enabling efficient attack investigation from system monitoring data. / Kulkarni, Sanjeev R.; Mittal, Prateek.

Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018. USENIX Association, 2020. p. 113-125 (Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - AIQL

T2 - Enabling efficient attack investigation from system monitoring data

AU - Kulkarni, Sanjeev R.

AU - Mittal, Prateek

PY - 2020/1/1

Y1 - 2020/1/1

N2 - The need for countering Advanced Persistent Threat (APT) attacks has led to the solutions that ubiquitously monitor system activities in each host, and perform timely attack investigation over the monitoring data for analyzing attack provenance. However, existing query systems based on relational databases and graph databases lack language constructs to express key properties of major attack behaviors, and often execute queries inefficiently since their semantics-agnostic design cannot exploit the properties of system monitoring data to speed up query execution. To address this problem, we propose a novel query system built on top of existing monitoring tools and databases, which is designed with novel types of optimizations to support timely attack investigation. Our system provides (1) domain-specific data model and storage for scaling the storage, (2) a domain-specific query language, Attack Investigation Query Language (AIQL) that integrates critical primitives for attack investigation, and (3) an optimized query engine based on the characteristics of the data and the semantics of the queries to efficiently schedule the query execution. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 857 GB of real system monitoring data (containing 2.5 billion events). Our evaluations on a real-world APT attack and a broad set of attack behaviors show that our system surpasses existing systems in both efficiency (124x over PostgreSQL, 157x over Neo4j, and 16x over Greenplum) and conciseness (SQL, Neo4j Cypher, and Splunk SPL contain at least 2.4x more constraints than AIQL).

AB - The need for countering Advanced Persistent Threat (APT) attacks has led to the solutions that ubiquitously monitor system activities in each host, and perform timely attack investigation over the monitoring data for analyzing attack provenance. However, existing query systems based on relational databases and graph databases lack language constructs to express key properties of major attack behaviors, and often execute queries inefficiently since their semantics-agnostic design cannot exploit the properties of system monitoring data to speed up query execution. To address this problem, we propose a novel query system built on top of existing monitoring tools and databases, which is designed with novel types of optimizations to support timely attack investigation. Our system provides (1) domain-specific data model and storage for scaling the storage, (2) a domain-specific query language, Attack Investigation Query Language (AIQL) that integrates critical primitives for attack investigation, and (3) an optimized query engine based on the characteristics of the data and the semantics of the queries to efficiently schedule the query execution. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 857 GB of real system monitoring data (containing 2.5 billion events). Our evaluations on a real-world APT attack and a broad set of attack behaviors show that our system surpasses existing systems in both efficiency (124x over PostgreSQL, 157x over Neo4j, and 16x over Greenplum) and conciseness (SQL, Neo4j Cypher, and Splunk SPL contain at least 2.4x more constraints than AIQL).

UR - http://www.scopus.com/inward/record.url?scp=85077453723&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85077453723&partnerID=8YFLogxK

M3 - Conference contribution

T3 - Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018

SP - 113

EP - 125

BT - Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018

PB - USENIX Association

ER -

Kulkarni SR, Mittal P. AIQL: Enabling efficient attack investigation from system monitoring data. In Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018. USENIX Association. 2020. p. 113-125. (Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018).