Detecting kernel-level rootkits using data structure invariants

TY - JOUR

T1 - Detecting kernel-level rootkits using data structure invariants

AU - Baliga, Arati

AU - Ganapathy, Vinod

AU - Iftode, Liviu

N1 - Funding Information: The authors thank Joe Kilian for insightful discussions about this work. They also thank the anonymous reviewers of this paper for their insightful comments. This work has been supported in part by the US National Science Foundation (NSF) under grants 0728937, 0831268, 0915394, and 0931992 and award from the Rutgers DIMACS center. This paper is a revised and expanded version of work published in the 24th Annual Computer Security Applications Conference [8].

PY - 2011

Y1 - 2011

N2 - Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify noncontrol data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits. This paper presents a novel technique to detect rootkits that modify both control and noncontrol data. The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously known rootkits, including those that modify noncontrol data structures.

AB - Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify noncontrol data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits. This paper presents a novel technique to detect rootkits that modify both control and noncontrol data. The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously known rootkits, including those that modify noncontrol data structures.

KW - Kernel-level rootkits

KW - invariant inference

KW - noncontrol data attacks

KW - static and dynamic program analysis

UR - http://www.scopus.com/inward/record.url?scp=79960552201&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960552201&partnerID=8YFLogxK

U2 - https://doi.org/10.1109/TDSC.2010.38

DO - https://doi.org/10.1109/TDSC.2010.38

M3 - Article

VL - 8

SP - 670

EP - 684

JO - IEEE Transactions on Dependable and Secure Computing

JF - IEEE Transactions on Dependable and Secure Computing

SN - 1545-5971

IS - 5

M1 - 5551160

ER -