Reverse Engineering Malware

January 20, 2019  |  Kate Brew

The Alien Labs team does a lot of malware analysis as a part of their security research. I interviewed a couple members of our Labs team, including Patrick Snyder, Eddie Lee, Peter Ewane and Krishna Kona, to learn more about how they do it.

Here are some of the approaches and tools and techniques they use for reverse engineering malware, which may be helpful to you in your own malware hunting endeavors. Please watch the webcast they did recently with Javvad Malik on reverse engineering malware and hear details and examples of how the Labs team investigated OceanLotus, PowerWare and Linux malware in recent situations.

Approaches in reverse engineering a malware sample

  • Reverse engineer: The most obvious approach is to completely reverse engineer a piece of malware. This obviously takes a great amount of time, so other approaches are more practical.
  • Exploitation techniques: Another approach you can take is to focus on the exploitation techniques of a piece of malware. Occasionally you will see a piece of malware that is using a new exploitation technique, or is exploiting a zero-day vulnerability. In this case you may be interested only in the specific exploitation technique so you can timebox your analysis and only look at the exploitation mechanisms.
  • Obfuscation: Malware will often obfuscate itself and make itself difficult to analyze. You might come across malware that you have seen before without obfuscation. In that case you may only want to focus on reverse engineering the new parts.
  • Encryption methods: A common type of malware these days is ransomware. Ransomware essentially encrypts the victim's files and locks them up so that they can't be accessed or read. Oftentimes the authors of ransomware will make mistakes when they implement the encryption mechanisms. So if you focus your research on the encryption mechanisms you might be able to find weaknesses in their implementation and/or you might be able to find hard-coded keys or weak algorithms.
  • C&C communication: This is something that is pretty commonly done when looking at malware. Analysts often want to figure out what the communication protocol is between a piece of malware on the client's side and the server on the command and control side. The communication protocol can actually give you a lot of hints about the malware’s capabilities.
  • Attribution: Murky area - kind of like a dark art. It usually involves a lot of guesswork, knowledge of malicious hacking teams and looking at more than one piece of malware.
  • Categorization and clustering: You can reverse engineer malware from a broader point of view. This involves looking at malware in bulk and doing a broad-stroke analysis on lots of different malware, rather than doing a deep dive.


Now, let’s look at techniques that can be utilized while analyzing malware.

  • First of all, we use static analysis. This is the process of analyzing malware or binaries without actually running them. It can be as simple as looking at metadata from a file. It can range from doing disassembly or decompilation of malware code to symbolic execution, which is something like virtual execution of a binary without actually executing it in a real environment.
  • Conversely, dynamic analysis is the process of analyzing a piece of malware when you are running it in a live environment. In this case, you are often looking at the behavior of the malware and looking at the side effects of what it is doing. You are running tools like process monitor and sysmon to see what kinds of artifacts a piece of malware produces after it is run.
  • We also use automated analysis. Oftentimes if you are looking at malware you want to automate things just to speed up the process to save time. However, use caution, as with automated analysis sometimes things get missed because you are trying to do things generically.
  • If a piece of malware contains things like anti-debugging routines or anti-analysis mechanisms, you may want to perform a manual analysis. You need to pick the right tools for the job.


  • IDA Pro is a really good tool for analyzing various samples of malware with diverse backgrounds. It also has a good add-on called HEX Rays Decompiler, which is a tool that can convert assembly language into more easily read pseudocode. It can help you in understanding the functionality of the code more quickly than looking at assembly language. When you open a sample in IDA Pro, you see the entry point of the malware. It has a graph view as well and you can switch between both hex code and the graph view. It will give you a quick representation of the mapping of the flow of execution as well. It has an SDK you can use if you like to develop plug-ins and automate and extract some of the useful information. IDA Pro also has a Python API that you can use if you prefer Python. The tool also has debugging functionality, but mainly it is used for static reverse engineering of malware.

Here’s IDA Pro:

reverse engineering malware - IDA Pro

Here’s PEiD:

reverse engineering malware - PeID

  • Other tools that we use include strings, file, and otool. They help us initially find the platform of the sample. If you look at some snapshots of these tools, they can tell you where the entry point of that sample is, what section, and if the sample is packed. They can also detect more than a hundred packers and can detect decrypters and compilers as well.

Here’s the file utility:

Generally, when we get a bunch of samples or an archive of samples from open-source feed, we use a file utility to find out if the file is a regular executable or for a Windows platform or OSX or Linux, or is it just a text file or a script.

  • Immunity Debugger is another popular debugger. If you open a sample in Immunity Debugger, it will give you an alert saying that the sample is packed and be asked if you want to proceed with the analysis. If you continue the analysis, you see an entry point, which is where it pushes all of the registers to stack. You can use this debugger to step through the execution of the sample to see the unpacked sample in memory. You can continue analyzing the samples step-by-step and use the debuggers in the tool for finding the malware's activities and the effects it has on the system.

Here’s Immunity Debugger:

reverse engineering malware Immunity Debugger

reverse engineering malware Immunity Debugger

For capturing network traffic, we use Wireshark or TCPDump. Here's a nice TCPDump cheatsheet.

For monitoring the activity on the system, we use system monitor and Regshot.

Sandboxes are another important step in reverse engineering malware, as often there are functionalities malware doesn't exhibit unless it is running in a suitable environment. One sandbox, malwr, comes from the people who built Cuckoo Sandbox. With malwr, you submit a sample and run it inside a VM. You can then run various dynamic analysis tools and static analysis tools referenced above and turn this into a nice, readable report.

Here is malwr:

reverse engineering malware tools malwr

  • Another Sandbox that is relatively new is Hybrid-Analysis. It is made by Payload Security and it functions very similar to malwr, but they have some of their own custom sandboxes running that may or may not be based on Cuckoo.

Here is Hybrid-Analysis:

reverse engineering malware hybrid analysis tool

Another major Sandbox tool for identifying malware is VirusTotal. VirusTotal is owned by Google, and they arguably have the biggest repository of both malware and known file types in general layout. If you are looking for any particular malware, it typically shows up in VirusTotal.

Here is VirusTotal:

VirusTotal for reverse engineering malware

Another new contender is DeepViz. DeepViz is being developed very actively, with new features on a regular basis. DeepViz functions very similarly to other Sandboxes, but sometimes it is beneficial to submit the same sample to multiple sandboxes to see if the behavior matches up or if it reacts differently.

Here is DeepViz:

DeepViz for reverse engineering malware

Which brings us to Cuckoo. Cuckoo is a malware analysis system. It contains many different tools, including some of the dynamic and static analysis tools that we mentioned earlier. Also, it is free. While other sandboxes are free, you are sharing your data by using them. If you set up Cuckoo on your own system you can keep everything localized and keep it to yourself, especially if you are analyzing something you don't want the world to know about yet.

Here is Cuckoo:

reverse engineering malware with cuckoo

Open Threat Exchange (OTX) is another key component we use in malware analysis.

To find out more about OTX there is a documentation center. You can also see information on our forums. There is a section specifically for OTX where you can see pulses. Also, just a few weeks ago we announced some enhancements to the OTX API. If you are a blogger, please note you can now embed pulses. So if you write a blog, you can just simply embed it within so users can read it and directly download the IoCs and other information. Read more.

Connecting OTX to your USM platform helps you to manage risk better and effectively take action on threats. A free trial of AlienVault USM is available.

Share this with others

Featured resources



2024 Futures Report

Get price Free trial