Anti-debugging and anti-VM techniques and anti-emulation [updated 2019]

These days malware is becoming more advanced. Malware Analysts use lots of debugging software and applications to analyze malware and spyware. Malware authors use some techniques to detect the presence of automatic analysis systems such as debuggers and Virtual Machines. In this article we will explore some of these commonly used techniques and practices to evade malware debugging software and sandboxes.

Tools required:

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

• Immunity debugger
• C/C++ compiler (msvc or GCC)
• Virtual Machine (Vmware of Vbox)

Introduction to debuggers

A Debugger is a piece of software used to analyze and instrument executable files. In order to analyze and intercept machine code debuggers use system calls and API commonly provided by the operating system. To intercept a single block of code, debuggers use a single stepping operation which can be turned on by setting the TRAP Flag in EFLAGS register. Debuggers use many types of breakpoints in order to stop at a particular memory address. The following are the type of breakpoints debuggers use.

1. Software Breakpoint.
2. Hardware breakpoint.
3. Memory breakpoints.
4. Conditional Breakpoints.

Software Breakpoints are the type of breakpoints where a debugger replaces the original instruction with an INT 0xcc instruction, which raises a software breakpoint interrupt routine and is returned back to the debugger to handle it. In an immunity debugger you can view your software breakpoint by pressing ALT + b

Breakpoints:

[plain]

Address Module Active Disassembly Comment

00401FF0 extracto Always JE SHORT extracto.00401FF7

00401FFC extracto Always MOV EBP,ESP

0040200A extracto Always CALL DWORD PTR DS:[<&KERNEL32.ExitPr

[/plain]

Hardware breakpoints use four of the debug register provided by the process in-order to incept at a particular breakpoint. These registers include DR0, DR1, DR2, DR3

We then flip the appropriate bits in the DR7 register to enable the breakpoint and set its type and length.

After the hardware breakpoint has been set and is reached the OS raises an INT 1 interrupt the single stepping event.

Debuggers then set up appropriate handlers to catch those exceptions.

Memory breakpoint:

In memory the breakpoint we use guard pages to set up a handler and if that page is accessed an exception handler is called.

Debuggers support many types of memory breakpoints

1. memory breakpoint on BYTE access.
2. memory breakpoint on WORD access.
3. memory breakpoint on DWORD access.

Conditional breakpoints:

Conditional breakpoints are managed by the debugger, and they are presented to users only if certain conditions are met.

For example you can set up conditional breakpoints in an immunity debugger which has the following syntax:

CONDITION = [ESP] = 0x0077ff89

Which will only be caught if the value pointed at the top of the stack is 0x0077ff89.

Memory breakpoints are only useful when you want to monitor calls to specific API with only certain parameters.

Debugging API on windows

Windows by default provides an API for debugging which is utilized by debuggers to debug applications. The API provided by windows is known as windows debugging API.

The following is a sample code to debug an application using windows debugging API.

[plain]

void EnterDebugLoop(const LPDEBUG_EVENT DebugEv)

{

DWORD dwContinueStatus = DBG_CONTINUE; // exception continuation

char buffer[100];

CONTEXT lcContext;

for(;;)

{

// Wait for a debugging event to occur. The second parameter indicates

// that the function does not return until a debugging event occurs.

WaitForDebugEvent(DebugEv, INFINITE);

// Process the debugging event code.

switch (DebugEv->dwDebugEventCode)

{

case EXCEPTION_DEBUG_EVENT:

// Process the exception code. When handling

// exceptions, remember to set the continuation

// status parameter (dwContinueStatus). This value

// is used by the ContinueDebugEvent function.

switch(DebugEv->u.Exception.ExceptionRecord.ExceptionCode)

{

case EXCEPTION_ACCESS_VIOLATION:

// First chance: Pass this on to the system.

// Last chance: Display an appropriate error.

break;

case EXCEPTION_BREAKPOINT:

if (!fChance)

{

dwContinueStatus = DBG_CONTINUE; // exception continuation

fChance = 1;

break;

}

lcContext.ContextFlags = CONTEXT_ALL;

GetThreadContext(pi.hThread, &lcContext);

ReadProcessMemory(pi.hProcess , (LPCVOID)(lcContext.Esp ),(LPVOID)&rtAddr, sizeof(void *), NULL );

if (DebugEv->u.Exception.ExceptionRecord.ExceptionAddress == pEntryPoint)

{

printf("n%sn", "Entry Point Reached");

WriteProcessMemory(pi.hProcess ,DebugEv->u.Exception.ExceptionRecord.ExceptionAddress,&OrgByte, 0x01, NULL);

lcContext.ContextFlags = CONTEXT_ALL;

GetThreadContext(pi.hThread, &lcContext);

lcContext.Eip--; // Move back one byte

SetThreadContext(pi.hThread, &lcContext);

FlushInstructionCache(pi.hProcess,DebugEv->u.Exception.ExceptionRecord.ExceptionAddress,1);

dwContinueStatus = DBG_CONTINUE ; // exception continuation

putBP();

break;

}

// First chance: Display the current

// instruction and register values.

break;

case EXCEPTION_DATATYPE_MISALIGNMENT:

// First chance: Pass this on to the system.

// Last chance: Display an appropriate error.

dwContinueStatus = DBG_CONTINUE ;

break;

case EXCEPTION_SINGLE_STEP:

printf("%s", "Single stepping event ");

dwContinueStatus = DBG_CONTINUE ;

break;

case DBG_CONTROL_C:

// First chance: Pass this on to the system.

// Last chance: Display an appropriate error.

break;

default:

// Handle other exceptions.

break;

}

break;

case CREATE_THREAD_DEBUG_EVENT:

//dwContinueStatus = OnCreateThreadDebugEvent(DebugEv);

break;

case CREATE_PROCESS_DEBUG_EVENT:

printf("%s", GetFileNameFromHandle(DebugEv->u.CreateProcessInfo.hFile));

break;

case EXIT_THREAD_DEBUG_EVENT:

// Display the thread's exit code.

//dwContinueStatus = OnExitThreadDebugEvent(DebugEv);

break;

case EXIT_PROCESS_DEBUG_EVENT:

// Display the process's exit code.

return;

//dwContinueStatus = OnExitProcessDebugEvent(DebugEv);

break;

case LOAD_DLL_DEBUG_EVENT:

char *sDLLName;

sDLLName = GetFileNameFromHandle(DebugEv->u.LoadDll.hFile);

printf("nDLl Loaded = %s Base Address 0x%pn", sDLLName, DebugEv->u.LoadDll.lpBaseOfDll);

//dwContinueStatus = OnLoadDllDebugEvent(DebugEv);

break;

case UNLOAD_DLL_DEBUG_EVENT:

// Display a message that the DLL has been unloaded.

//dwContinueStatus = OnUnloadDllDebugEvent(DebugEv);

break;

case OUTPUT_DEBUG_STRING_EVENT:

// Display the output debugging string.

//dwContinueStatus = OnOutputDebugStringEvent(DebugEv);

break;

case RIP_EVENT:

//dwContinueStatus = OnRipEvent(DebugEv);

break;

}

// Resume executing the thread that reported the debugging event.

ContinueDebugEvent(DebugEv->dwProcessId,

DebugEv->dwThreadId,

dwContinueStatus);

}

}

int main(int argc ,char **argv)

{

DEBUG_EVENT debug_event = {0};

STARTUPINFO si;

FILE *fp = fopen(argv[1], "rb");

ZeroMemory( &si, sizeof(si) );

si.cb = sizeof(si);

ZeroMemory( &pi, sizeof(pi) );

CreateProcess ( argv[1], NULL, NULL, NULL, FALSE,

DEBUG_ONLY_THIS_PROCESS, NULL,NULL, &si, &pi );

printf("Passed Argument is %sn", OrgName);

pEntryPoint = GetEP(fp); // GET the entry Point of the Application

fclose(fp);

ReadProcessMemory(pi.hProcess ,pEntryPoint, &OrgByte, 0x01, NULL); // read the original byte at the entry point

WriteProcessMemory(pi.hProcess ,pEntryPoint,"xcc", 0x01, NULL); // Replace the byte at entry point with int 0xcc

EnterDebugLoop(&debug_event); // User-defined function, not API

return 0;

}

int main(int argc ,char **argv)

{

DEBUG_EVENT debug_event = {0};

STARTUPINFO si;

FILE *fp = fopen(argv[1], "rb");

ZeroMemory( &si, sizeof(si) );

si.cb = sizeof(si);

ZeroMemory( &pi, sizeof(pi) );

CreateProcess ( argv[1], NULL, NULL, NULL, FALSE,

DEBUG_ONLY_THIS_PROCESS, NULL,NULL, &si, &pi );

printf("Passed Argument is %sn", OrgName);

pEntryPoint = GetEP(fp); // GET the entry Point of the Application

fclose(fp);

ReadProcessMemory(pi.hProcess ,pEntryPoint, &OrgByte, 0x01, NULL); // read the original byte at the entry point

WriteProcessMemory(pi.hProcess ,pEntryPoint,"xcc", 0x01, NULL); // Replace the byte at entry point with int 0xcc

EnterDebugLoop(&debug_event); // User-defined function, not API

return 0;

}

[/plain]

 

Anti-debugging techniques

Now in order to frustrate the malware analyst, malware can be detected in the presence of debuggers and show up in unexpected events. In order to detect the presence of a debugger, malware can either read some values or it can use API present to detect if the malware is being debugged or not.

One of the simple debugger detection tricks includes using the winAPI function known as KERNEL32.IsDebuggerPresent.

[plain]

#define WIN32_LEAN_AND_MEAN

#include <windows.h>

#include <stdio.h>

int main(int argc, char **argv)

{

if (IsDebuggerPresent())

{

MessageBox(HWND_BROADCAST, "Debugger Detected", ""Debugger Detected"", MB_OK);

exit();

}

MessageBox(HWND_BROADCAST, "Debugger Not Detected", ""Debugger Not Detected"", MB_OK);

return 0;

}

[/plain]

Want more information on anti-debugging, check out this article!

Detecting a debugger using PEB:

When the process is created using CreateProcess API, and if the creation flag is set as DEBUG_ONLY_THIS_PROCESS then a special field is set in the PEB data structure in the memory

[plain]

#define WIN32_LEAN_AND_MEAN

#include <windows.h>

#include <stdio.h>

int __naked detectDebugger()

{

__asm

{

ASSUME FS:NOTHING

MOV EAX,DWORD PTR FS:[18]

MOV EAX,DWORD PTR DS:[EAX+30]

MOVZX EAX,BYTE PTR DS:[EAX+2]

RET

}

}

int main(int argc, char **argv)

{

if (detectDebugger())

{

MessageBox(HWND_BROADCAST, "Debugger Detected", ""Debugger Detected"", MB_OK);

exit();

}

MessageBox(HWND_BROADCAST, "Debugger Not Detected", ""Debugger Not Detected"", MB_OK);

return 0;

}

[/plain]

Detection using HEAP flags:

When a program is run under a debugger, and is created using the debug process creation flags. The heap flags are changed. These Flags exit at a different location depending upon the version of the operating system.

On Windows NT based systems these flags exist at 0x0c offset from heap base.

ON Windows Vista based systems and later they exist at location 0x40 offset from the heap base.

These two flags initialized are 'Force flags' and 'flags'.

ProcessHeap Base Points towards a _HEAP structure are defined as:

Reference :http://www.nirsoft.net/kernel_struct/vista/HEAP.html

[plain]

typedef struct _HEAP

{

HEAP_ENTRY Entry;

ULONG SegmentSignature;

ULONG SegmentFlags;

LIST_ENTRY SegmentListEntry;

PHEAP Heap;

PVOID BaseAddress;

ULONG NumberOfPages;

PHEAP_ENTRY FirstEntry;

PHEAP_ENTRY LastValidEntry;

ULONG NumberOfUnCommittedPages;

ULONG NumberOfUnCommittedRanges;

WORD SegmentAllocatorBackTraceIndex;

WORD Reserved;

LIST_ENTRY UCRSegmentList;

ULONG Flags;

ULONG ForceFlags;

ULONG CompatibilityFlags;

ULONG EncodeFlagMask;

HEAP_ENTRY Encoding;

ULONG PointerKey;

ULONG Interceptor;

ULONG VirtualMemoryThreshold;

ULONG Signature;

ULONG SegmentReserve;

ULONG SegmentCommit;

ULONG DeCommitFreeBlockThreshold;

ULONG DeCommitTotalFreeThreshold;

ULONG TotalFreeSize;

ULONG MaximumAllocationSize;

WORD ProcessHeapsListIndex;

WORD HeaderValidateLength;

PVOID HeaderValidateCopy;

WORD NextAvailableTagIndex;

WORD MaximumTagIndex;

PHEAP_TAG_ENTRY TagEntries;

LIST_ENTRY UCRList;

ULONG AlignRound;

ULONG AlignMask;

LIST_ENTRY VirtualAllocdBlocks;

LIST_ENTRY SegmentList;

WORD AllocatorBackTraceIndex;

ULONG NonDedicatedListLength;

PVOID BlocksIndex;

PVOID UCRIndex;

PHEAP_PSEUDO_TAG_ENTRY PseudoTagEntries;

LIST_ENTRY FreeLists;

PHEAP_LOCK LockVariable;

LONG * CommitRoutine;

PVOID FrontEndHeap;

WORD FrontHeapLockCount;

UCHAR FrontEndHeapType;

HEAP_COUNTERS Counters;

HEAP_TUNING_PARAMETERS TuningParameters;

} HEAP, *PHEAP;

[/plain]

Following the C program can be used to detect the presence of a debugger using heap flags

[c]

int main(int argc, char* argv[])

{

unsigned int var;

__asm

{

MOV EAX, FS:[0x30];

MOV EAX, [EAX + 0x18];

MOV EAX, [EAX + 0x0c];

MOV var,EAX

}

if(var != 2)

{

printf("Debugger Detected");

}

return 0;

}

[/c]

Virtual machine detection or emulation detection

Malware samples are usually analyzed by analysts in an isolated environment such as Virtual Machine. In order to thwart the analysis of samples inside a virtual machine malware include anti-vm protection or they simply exit when malware is run in an isolated environment.

The following techniques can be used to detect if a sample is running inside a VM.

1. Timing Based.
2. Artifacts based.

Timing based detection

"The Time Stamp Counter (TSC) is a 64-bit register present on all x86 processors since the Pentium. It counts the number of cycles since reset". (Wikipedia)

If the code is being emulated then, there will be change in the time stamp between.

The Result in stored in EDX:EAX format

Now the time difference in a real host machine would be usually less than 100, but if the code is emulated the difference will be huge.

[c]

int main(int argc, char* argv[])

{

unsigned int time1 = 0;

unsigned int time2 = 0;

__asm

{

RDTSC

MOV time1,EAX

RDTSC

MOV time2, EAX

}

if ((time2 - time1) > 100)

{

printf("%s", "VM Detected");

return 0;

}

printf("%s", "VM not present");

return 0;

}

[/c]

The above program uses time stamp instruction to detect the presence of Virtual Machine.

Artifact based detection

Malwares leverage on the presence of Virtual Machine configuration based on file, network or device artifacts. Malwares usually check the presence of these artifacts to detect the presence of a debugger or Virtual Environment.

The best case would be registry artifacts, Vmware creates registry keys for Virtual Disk Controller, which can be located in registry using the following key.

HKLMSYSTEMCurrentControlSetServicesDiskEnum�

as "SCSIDisk&Ven_VMware_&Prod_VMware_Virtual_S&Rev_1.04&XXX&XXX"

[c]

int main(int argc, char **argv)

{

char lszValue[100];

HKEY hKey;

int i=0;

RegOpenKeyEx (HKEY_LOCAL_MACHINE, "SYSTEMCurrentControlSetServicesDiskEnum", 0L, KEY_READ , &hKey);

RegQueryValue(hKey,"0",lszValue,sizeof(lszValue));

printf("%s", lszValue);

if (strstr(lszValue, "VMware"))

{

printf("Vmware Detected");

}

RegCloseKey(hKey);

return 0;

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

}

[/c]