Due to the various features of Voice over Internet Protocol (VoIP), this technol- ogy has attracted the attention of many users in comparison with the tradi- tional telephony system. However, with the growth of this technology, the security issues and protection of its users against different kinds of threats have been raised as an essential requirement. Session Initiation Protocol is a pre- dominant protocol to initiate and terminate multimedia sessions in VoIP net- works that provide simplicity and text ‐ based features. Despite its mentioned advantages, these features impose several vulnerabilities on VoIP networks. Denial of Service attack, as one of the most common attacks against VoIP net- works, is also a noted security issue in the Internet Protocol platforms. In such attacks, the attacker tries to prevent service from authorized users by consum- ing server resources. These attacks can be launched by sending out ‐ of sequence messages, malformed messages, and flooding different kinds of messages. In this study, a new anomaly based method is presented for detection and preven- tion of these attacks. Normal traffic of a VoIP network is modeled by making a finite state machine, which is used for attack detection besides other proposed modules. Furthermore, a whitelist method is implemented using Bloom data structure for attack prevention. The proposed method is completely imple- mented and tested using different test scenarios. The obtained results show that by using proposed method, attacks can be detected more accurately with lower false ratios and delay in comparison with the existing methods.

Keywords