With smarter intrusion algorithms of the polymorphic kind, the protection provided by intrusion detection systems often proves to be ineffective. This paper addresses some malicious morphing strategies, the difficulties in confronting them, and possible countermeasures that can be effective.

Buffer overflows are used to inject malicious code in shellcode polymorphic attacks. The injected code has three components: the NOP sled, the payload or actual shellcode, and the return address zone. “An existing address value on the stack [is overwritten] with values contained within the return zone [... causing] the control flow to jump back into the input string” of the attacker.

Long sequences of NOP and return addresses help in the detection of normal shellcodes, so obfuscation is used to prevent detection. Two obfuscation techniques are common: rewriting codes automatically so that instances retain the semantics but are different in syntax, and encrypting the shellcode with a randomly chosen key. A “semantically prepended” decoding routine decrypts the code before the payload is triggered.

Smart polymorphism makes shellcode attacks very dangerous. This paper quantitatively analyzes “the strengths and limitations of shellcode polymorphism [... for both] encryption-based evasion methods and targeted ‘blending’ attacks.” It provides interesting metrics to measure the effectiveness of polymorphic engines, gives insight into their design, and suggests how to defend against them. This work has been positively referred to in other recent research [1,2].