Stacks and Heap

Memory is a crucial resource for any system when conducting reverse engineering. Malware analysts must understand the way memory is assigned to a program.

During the execution stage of a program, the memory that is assigned can be divided into four segments: stack, heap, text and data.

Let’s take a look at memory stacks and heap in C code and how they’re used.

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

What is a stack?

A stack is a container of elements that are inserted and removed according to the last in, first out (LIFO) principle in order to define data structure. It can be thought of as a stack of kitchen plates. The most recently reserved element or last element added is always the first to be accessed, freed or operated upon. Elements can only be added or removed from the top of the stack. In other words, only the top element can be accessed or operated upon. Insertions and deletions are only permitted at one end of the list of elements in a stack.

This feature of stacks which allows elements to be added or removed from only one end is known as linear data structure.

Stacks are a very special area of a computer’s memory that stores or allocates local variables created by a function. The memory allocations for these variables are automatically erased or de-allocated once the computing task is completed. Here, the allocation and deallocation are automatically done by compiler instructions.

Stack operations

There are basically two operations that can be performed on a stack. They are push and pop. A variable called TOP, associated with all stacks, is used to store the address of the topmost element available in the stack. Another variable called MAX stores the maximum number of elements held in a stack. A stack without elements, an empty stack, has the condition TOP = null. When the stack is full, it has the condition TOP = MAX -1.

Push operation

As shown, the push operation adds a new element to the topmost position of the stack and increases the value of the top by a value of 1. Before adding a new element, the stack needs to be checked if full, TOP = MAX -1. A push operation performed on a full stack will result in the printing of an OVERFLOW message.

Below is a code to implement push in C.

void push (int value) 

{

if (TOP == (MAX-1)) 

{ Printf ("New element couldn't be pushed onto stack. Stack is full.\n"); }

else

{

TOP=TOP+1;

stack[TOP] = value;

}

}

PoP operation

We see that in a pop operation, the topmost element is deleted or removed from the stack and the TOP variable is decreased by a value of 1.

Below is a code to implement pop in C.

int pop (int value) 

{

If (TOP==(MAX-1)) 

{ value = stack [ TOP ];

TOP = TOP - 1;   

return value; }

else {

Printf ( "PoP operation couldn't be performed, Stack is empty.\n" );

   }

}

What is a heap?

A heap is a managed memory region that allows for the dynamic allocation of variable-sized blocks of memory in runtime.   Basically, a heap is the area within memory where global variables are stored.

Unlike a stack, the variables in a heap must be manually created and destroyed. Data sets in a heap are allocated randomly, unlike a stack where they follow a LIFO pattern. This results in a scattered memory allocation. Unlike a stack where allocation and deallocation are affected through compiler instructions, heap allocations and deallocations are done by the programmer. 

A condition called memory leak occurs when a programmer fails to free memory or deallocate it after all allocations have been done.

malloc( ) and free( ) functions

The malloc() function is used to allocate or reserve a block of memory in no definite order. The allocation is done according to the specified number of bytes. The format is given as:

ptr = (type *) malloc(size);

The pointer ptr stores the address of the allocated chunk of memory. size is the required amount of storage. Even though this is measured in bytes, care must be taken to reserve enough space required for the variable type.

Example 1 : ptr = (float*) malloc(110 * sizeof(float));

Here, 440 bytes of memory are allocated because the size of a float is four bytes.

After allocations are done, memory space must be freed up to prevent leaks. The function free() is used to free up allocated space in memory for which address is stored at ptr.

Example 2 : free(ptr);

In this example, the free function is used to free the memory pointed by ptr in example 1.

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

Memory stacks in reverse engineering

Reverse engineering enthusiasts and malware analysts without a solid understanding of memory stacks will find it difficult to deal with various types of problems as they’re mostly stack-based. With the recent significant increase in heap spray attacks, malware enthusiasts need to take a second look at memory heaps to bridge the race with malware developers.

Malware analysis and reverse engineering is not an easy path as it partly includes digital forensics and programming. To succeed, one needs to take at least these two desirable certifications: Certified Information Systems Security Professional (CISSP) and Certified Ethical Hacker (CEH).  In addition, training courses can help you better understand these difficult topics, such as the Malware Analysis and Reverse Engineering Learning Path in Infosec Skills. The courses can be found Here.  

Sources

1. http://www.ijirt.org/master/publishedpaper/IJIRT101357_PAPER.pdf
2. Reversing: Secrets of Reverse Engineering By Eldad Eilam