Memory Forensics for Virtualized Hosts

Detecting In-Memory Malware Threats

Memory analysis plays a key role in identifying sophisticated malware in both user space and kernel space, as modern threats are often file-less, operating without creating a file system artifact.

The most effective approach to the detection of these sophisticated malware components is to install on the protected operating system an agent that continuously monitors the OS memory for signs of compromise. However, this approach has a number of drawbacks. First, the agent introduces a constant overhead in the monitored OS — caused by both the resources used by the agent process (e.g., CPU, memory) and the instrumentation used to capture relevant events (e.g., API hooking). Second, a malware sample can detect the presence of an agent and attempt to either disable the agent or evade detection. Third, depending on how it is deployed, the agent not have access to specific portions of the user-space and kernel-space memory, and, as a consequence, may miss important evidence of a compromise. Finally, deploying, maintaining, and updating agents on every endpoint can be challenging, especially in heterogeneous deployments where multiple versions of different operating systems and architectures coexist.

A complementary approach to the detection of these sophisticated threats relies on off-line analysis of an endpoint’s memory dump, which is a snapshot of the contents of volatile memory at a specific point in time. In most cases, this type of forensic analysis is initiated manually by a security administrator when specific evidence needs to be collected and preserved. The process of analyzing these memory dumps is often referred to as memory forensics. Memory forensics focuses on extracting meaningful data from the unstructured stream of bytes contained in a memory dump — a process often referred to as “closing the semantic gap.”

Memory forensics has been long considered a challenging process for a number of reasons. On the one hand, the simple process of collecting a memory dump can be hindered by the lack of introspection mechanisms (for example, in embedded systems). On the other, the dynamic nature of memory management systems can create race conditions that result in an inconsistent memory dump (or memory smearing).

A number of ad hoc memory forensics techniques have been proposed in the past, and only very recently have academic researchers provided a methodical evaluation of memory forensics approaches that highlights the benefits and the costs of different techniques [1].

Existing memory forensics techniques can be roughly divided into two classes: dynamic approaches and static approaches. Dynamic approaches analyze the memory in vivo, while the underlying operating system is executing. These techniques have the advantage of allowing for a timely identification of security issues. However, the performance cost, the complexity of analyzing a moving target, and the risk of introducing instability often make these techniques non-applicable to mission-critical systems.

Static approaches operate on a static memory dump of a system, taken at a specific point in time. These approaches do not interfere with the run-time operation of the system, except for the interval during which the dump is recorded. However, evidence of a threat must be present in memory when the dump is collected. In addition, malicious behavior related to a series of events may not be detectable [2].

Memory Forensics and Virtualization

Memory acquisition is one of the most critical steps in the memory forensics process, and it is based on the premise that it is possible to acquire a running system’s memory. While memory acquisition might be challenging in several operational contexts, it is seamless in virtualized environments in which a hypervisor is responsible for tracking and managing the virtual memory of multiple guest operating systems.

In fact, most hypervisors provide programmatic interfaces for suspending and restarting guest machines, a functionality that requires saving a guest’s state to disk so that it can be restored at a later time.

The capability to save the machine state to disk creates a natural synergy between virtualization and memory forensics. In some cases, it is even possible to collect the memory dump of a running virtualized machine without needing to interrupt its execution.

For example, VMware’s ESX provides a mechanism for the creation of machine snapshots which include the complete state of the system (e.g., their virtual disks, virtual network interfaces, and, of course, the virtual memory). The guest machine’s configuration is saved into a file with a “.vmss” extension, and the guest machine’s memory is saved into a separate file, which has a “.vmem” extension.

The highest cost in snapshotting is the time required to write the guest machine’s memory to disk, which can be quite significant for a virtual machine with a large memory footprint. The VMware virtualization infrastructure provides a lazy snapshot mechanism, enabling the guest machine to continue its execution while its state is being saved. In lazy snapshotting, a trace flag is added to each page in the guest memory. During the guest machine’s execution, when a write operation is performed on a page with the flag, that page is saved to disk before the guest machine modifies that page. After the page is saved to disk, the write trace flag is removed from the page, and then the guest is allowed to modify the page to continue execution. This mechanism significantly reduces the downtime for guest machines with large memory footprints.

Analyzing Memory Artifacts

Once the virtual memory snapshot of a virtualized host is collected, the next step is to analyze that memory snapshot and extract useful artifacts from the raw stream of bytes contained in the memory dump, such as the list of running processes, open network connections, drivers currently loaded in the kernel, etc.

This “reconstruction” is carried out with the help of memory analysis tools. One of the most commonly used tools is Volatility Framework [3], which supports the analysis of memory snapshots for Windows, Linux, and MacOS operating systems. The Volatility3 Framework can load .vmem files generated by VMware’s ESX during the memory snapshot process and is able to reconstruct the memory image of the machine for memory analysis.

The Volatility3 Framework supports memory forensics through three main components:

• Memory Layers:  Memory Layers allow one to access the data from the memory dump in a way that is independent of how the data is stored. Memory Layers implementss various memory-mapping algorithms to translate a memory address into the actual data from the memory dump, in the process sometimes performing various table lookups and data translations.
• Templates and Objects: Memory forensic analysis requires access to an operating system’s data structures. The instances of these data structures are called Objects. A Template contains all the information needed about the Object’s structure without being populated by any data. At the time of memory analysis, a Template is mapped on the memory layer at a specific offset to construct an Object, which then is used for the structural analysis of the memory.
• Symbol Tables: At compile-time, compilers may include meta-information about the data structures and variables used by a program (e.g., using a PDB file in Windows or a DWARF file in Linux). Volatility provides unified access to this information through Symbol Tables, which contain both the Templates for data structures used by the program and the addresses of specific data structures.

In short, Symbol Tables help in generating Templates: templates are mapped on the memory layer at a specific offset to construct the Objects. The memory analysis framework uses these Objects to perform structural analysis on the memory. Figure 1 shows the memory reconstruction and the forensic analysis process of the Volatility3 Framework.

Detecting Malware

Once the OS data structures and the objects have been identified, malware threats can be detected using a number of approaches. The most direct approach is the use of signatures that define specific Indicators of Compromise (IoC), such as specific portions of code or strings in memory. The most-used format for specifying these signatures is the YARA language [4].

YARA signatures can be easily applied to both data and code associated with the processes identified in a memory dump, and through the years the security community has developed a vast library of signatures for a large swath of threats. A major advantage of using YARA in the context of memory forensics is that the code of a threat is more likely to be unpacked and easily recognized when in memory (compared to when the code is packed in the corresponding executable file).

A complementary approach is based on the structural analysis of memory. In this case, several heuristics are used to determine the “health” of specific memory structures. For example, some rootkits use various techniques to hide their presence and avoid detection. The rootkit might remove itself from the registry of kernel modules or add hooks to various APIs used by the monitoring processes. It is possible to identify these evasive threats using memory analysis by directly accessing the OS objects and identifying “unaccounted” components or suspicious modifications to the function addresses of a process.

A third approach is based on the comparative analysis of similar memory images. In this case, the assumption is that many copies of the same virtualized operating system (e.g., the copies that are executed in a VDI deployment) have very similar internal structures, since they are all “cloned” from the same original image. Therefore, by comparing instances and identifying “black sheep” instances with unique and anomalous memory structures, it is possible to identify compromised hosts [5]. However, this approach assumes a homogenous system and requires a long and somewhat complex setup to compare in-memory data structures; therefore, it will not be discussed further here.

Examples

In-memory, file-less attacks are a type of stealth attack that evades detection by most security solutions and frustrates forensic analysis efforts based on the analysis of file system artifacts.

For example, the Netwalker file-less ransomware [6] leverages a reflective, dynamically linked library (DLL) injection technique, also referred to as “reflective DLL loading.” The technique allows for the injection of a DLL from memory rather than from disk. This technique is stealthier than regular DLL injection because, aside from not needing the actual DLL file on disk, it does not invoke the Windows loader. This eliminates the need to register the DLL as a loaded module of a process and evades DLL-loading monitoring tools.

In the following, we present the analysis of the specific sample with hash f4656a9af30e98ed2103194f798fa00fd1686618e3e62fba6b15c9959135b7be. The sample’s execution begins with a PowerShell script; the script hides a ransomware DLL under multiple encryption, obfuscation, and encoding layers. The script extracts the DLL and injects the DLL into the memory of a legitimate instance of explorer.exe.

In this case, we used ESX to collect a virtual machine snapshot after a successful Netwalker infection. Volatility3 has many useful plugins for malware analysis. One of the plugins, called MalFind, scans all the processes and lists all the memory ranges with read, write, and execute permission that potentially contain injected code. Figure 2 shows the output of the MalFind plugin when applied to the infected memory snapshot. The output lists all the memory ranges that potentially contain injected code, together with the associated process name and ID.

By using the dump option of the MalFind plugin, all the listed potential injected code was dumped into separate files. Then, we performed a YARA signature match to identify the presence of a PE file in the dumped files. As a result, a PE file was detected in the explorer.exe dump file. A PE file in a memory section with read, write, and execute permission is highly suspicious and requires more investigation. To identify the threat further, a YARA signatures match is applied to the entire explorer.exe process memory. The YARA signatures successfully identify the injected code as an instance of the Netwalker ransomware module based on the strings found in the process address space, as shown in Figure 3.

Memory analysis is not limited to identifying threats through pattern matching. For example, consider a threat that uses API hooking, a technique that is actively used to evade detection. API hooks can intercept the invocation of functions that are used to monitor the OS environment and change the data returned by the API. Malware threats often use API hooking to prevent popular monitoring applications like Process Explorer (procexp), ProcessHacker, Windows Task Manager, etc., from revealing the presence of the malware components.

For example, the Thanos ransomware [7] uses a tool, called ProcessHide, to hide its presence on an infected machine. ProcessHide takes as parameters the process IDs of the monitoring applications processes and the process name to hide. ProcessHide then hooks the NtQuerySystemInformation API call to hide the process from being listed by the monitoring applications.

To demonstrate how to detect and identify this threat using memory forensic analysis, we executed the Thanos ransomware sample with hash 58bfb9fa8889550d13f42473956dc2a7ec4f3abb18fd3faeaa38089d513c171f in an ESXi-hosted virtual machine. As shown in Figure 4, ProcessHacker can list the malware process “58bfb9fa88895.exe” easily.

The malware then uses the ProcessHide tool to hide its presence from the ProcessHacker tool. The successful execution of ProcessHide obscures the malware process, and the malware is no longer visible to ProcessHacker, as shown in Figure 5.

The infected virtual machine with the hidden process is then snapshotted, and the memory file is loaded in Volatility3 for memory analysis. Volatility3 reconstructs the memory image and creates OS objects using symbol tables and templates. In particular, Volatility3 provides a plugin, named PsList, which can list all the processes executing in the system. The plugin does this by first locating the PsActiveProcessHead pointer in the kernel’s memory image. This pointer points to a list of EPROCESS structures. The EPROCESS structure is the kernel’s representation of a process object. The EPROCESS structure contains a LIST_ENTRY structure to create a linked list of all running processes. The plugin traverses the linked list and lists all running processes in the system.

Since the plugin generates the process list by directly accessing the OS pointer and kernel objects, the hidden malware process “58bfb9fa88895.exe” is listed, as shown in Figure 6.

We developed a simple plugin for Volatility3 to detect API hooks. This plugin can detect the hooked functions of a process by analyzing its import table. The plugin first lists all the DLL loaded by the process and records the base address and the size of each DLL. Then the plugin analyzes the import table of the process binary image. Each import table entry and the Import Address Table (IAT) entries are verified. The IAT contains the addresses of the imported functions of the DLL. A valid function address in IAT should point to a memory location within the imported DLL memory range. If the plugin finds a function address that points to a memory location outside of the memory range of the DLL, then that function address is marked as a hook pointer. The plugin extracts the first few bytes of the hook code and disassembles it for further analysis. Figure 7 shows the output of the plugin when applied to the ProcessHacker.exe process. The plugin lists all imported functions and the corresponding addresses stored in the IAT. The plugin detects a hook on NtQuerySystemInformation, as the address of NtQuerySystemInformation points to a location (0x191664a0000) that is outside of the ntdll.dll memory range.

The plugin can also extract the entire hook module for further analysis, as shown in Figure 8.  The hook module acts as a proxy, and it calls the function NtQuerySystemInformation on behalf of the application. After the execution of NtQuerySystemInformation, control returns to the hook module. The hook module removes the process that is requested to hide from the process list returned by NtQuerySystemInformation. The process name to hide is stored at the end of the hook module. The plugin also extracts the hidden process name, which is “58bfb9fa88895.exe,” as shown in Figure 8.

Conclusions

Sophisticated malware threats require a complementary set of different approaches for efficient detection. A key tool in the workbench of the malware analyst is the use of memory forensics. As datacenter deployments increasingly leverage virtualization, there is a unique opportunity to combine the memory snapshot capabilities provided by hypervisors with the advanced memory analysis tools developed by the security community to detect and analyze sophisticated malware threats.

Bibliography

[1] F. Pagani and D. Balzarotti, “Back to the Whiteboard: a Principled Approach for the Assessment and Design of Memory Forensic Techniques,” in Proceedings of the USENIX Security Symposium, 2019. [2] A. Case and G. Richard, “Memory forensics: The path forward,” Digital Investigation: The International Journal of Digital Forensics & Incident Response, 2016. [3] Volatility, “Volatility 3,” 2021. [Online]. Available: https://volatility3.readthedocs.io/en/latest/. [4] VirusTotal, “YARA Documentation,” 2021. [Online]. Available: https://yara.readthedocs.io/en/stable/. [5] A. Bianchi, Y. Shoshitaishvili, C. Kruegel and G. Vigna, “Blacksheep: Detecting Compromised Hosts in Homogeneous Crowds,” in Proceedings of the ACM Conference on Computer and Communications Security (CCS), 1012. [6] K. Victor, “Reflective Loading Runs Netwalker Fileless Ransomware,” 2020. [Online]. Available: https://www.trendmicro.com/en_us/research/20/e/netwalker-fileless-ransomware-injected-via-reflective-loading.html. [7] K. Lu, Fortinet, [Online]. Available: https://www.fortinet.com/blog/threat-research/analysis-of-net-thanos-ransomware-supporting-safeboot-with-networking-mode.