Shellcode detector for malicious document hunting

Chong Kuan Chen, Shen Chieh Lan, Shiuhpyng Shieh

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations


Advanced Persistent Threat (APT) attacks became a major network threat in recent years. Among APT attack techniques, sending a phishing email with malicious documents attached is considered one of the most effective ones. Although many users have the impression that documents are harmless, a malicious document may in fact contain shellcode to attack victims. To cope with the problem, we design and implement a malicious document detector called Forensor to differentiate malicious documents. Forensor integrates several open-source tools and methods. It first introspects file format to retrieve objects inside the documents, and then automatically decrypts simple encryption methods, e.g., XOR, rot and shift, commonly used in malware to discover potential shellcode. The emulator is used to verify the presence of shellcode. If shellcode is discovered, the file is considered malicious. The experiment used 9,000 benign files and more than 10,000 malware samples from a well-known sample sharing website. The result shows no false negative and only 2 false positives.

Original languageEnglish
Title of host publication2017 IEEE Conference on Dependable and Secure Computing
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages2
ISBN (Electronic)9781509055692
StatePublished - 18 Oct 2017
Event2017 IEEE Conference on Dependable and Secure Computing - Taipei, Taiwan
Duration: 7 Aug 201710 Aug 2017

Publication series

Name2017 IEEE Conference on Dependable and Secure Computing


Conference2017 IEEE Conference on Dependable and Secure Computing


  • Malicious documents
  • Malware
  • Shellcode


Dive into the research topics of 'Shellcode detector for malicious document hunting'. Together they form a unique fingerprint.

Cite this