Unsupervised Learning of Network Traffic Behaviors for Insider Threat Detection

Insider threats are a costly and dangerous problem for government and non-government organizations alike. Considering an insider's inherently privileged level of access on a network, the main principle of network defense'keep potential threats and outsiders out'does not apply to insider threats. Current defenses are largely based on the detection of insider threat indicators and rely on up-to-date datasets. However, insider threat activity is innumerable and as diverse as human behavior itself. We hypothesize that characterizing and examining host and organization behavior demonstrated on a network presents an opportunity to circumvent this problem. Leveraging machine learning to detect behavioral anomalies that indicate the presence of an insider threat would enable network administrators to quickly locate and mitigate such threats before they cause serious damage. We demonstrate this methodology by developing a system that extracts host and organization behavior in three different ways from network traffic and uses population-relative metrics to determine host conformity with organizational norms. After testing the system on an operational network with over 8,000 hosts, we show through a series of case studies that our system is effective in detecting behavioral anomalies and that our behavior extraction methods are complementary.