Abstract
We consider the problem of identifying obscure chat-like botnet command and control (C & C) communications, which are indistinguishable from human-human communication using traditional signature-based techniques. Existing passive-behavior-based anomaly detection techniques are limited because they either require monitoring multiple bot-infected machines that belong to the same botnet or require extended monitoring times. In this paper, we explore the potential use of active botnet probing techniques in a network middle-box as a means to augment and complement existing passive botnet C & C detection strategies, especially for small botnets with obfuscated C & C content and infrequent C & C interactions. We present an algorithmic framework that uses hypothesis testing to separate botnet C & C dialogs from human-human conversations with desired accuracy and implement a prototype system called BotProbe. Experimental results on multiple real-world IRC bots demonstrate that our proposed active methods can successfully identify obscure and obfuscated botnet communications. A real-world user study on about one hundred participants also shows that the technique has a low false positive rate on human-human conversations. We discuss the limitations of BotProbe and hope this preliminary feasibility study on the use of active techniques in botnet research can inspire new thoughts and directions within the malware research community.
