Ethical hacking: Buffer overflow

When taking external input, an application needs to allocate memory to store that input. Many high-level programming languages will do this behind the scenes, but some languages (like C/C++) allow the programmer to allocate memory directly through functions like malloc.

A buffer overflow vulnerability occurs when the application tries to store more data in the allocated memory than there is room for. This can occur for a variety of reasons, including:

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

• Failing to check input length when reading
• Forgetting to allocate space for the null terminator
• Input lengths that cause an integer overflow

Regardless of the reason, if an application tries to write to memory beyond the range of its allocated buffer, this means that it is writing to the memory allocated for other purposes within the application. Due to the structure of how memory is allocated within a computer, this can be extremely useful to an attacker since it allows them to control the execution of the program.

Buffer overflow exploitation

Exploitation of a buffer overflow vulnerability is fairly simple. If a program incorrectly allocates memory for user input or insecurely reads data into that memory space, a buffer overflow vulnerability exists. This vulnerability can be exploited by a hacker simply by providing more input to the application than the allocated buffer is capable of holding.

Overflowing a buffer with meaningless or random input is likely to just cause a segmentation fault or an error in the program. However, the structure of the stack means that a well-designed buffer overflow exploit can do much more, allowing an attacker to control execution flow and run malicious code on the system.

The stack

An application can allocate memory on the stack or on the heap. The stack is commonly used for function arguments and local variables, and the heap stores dynamic memory (allocated using the new command in C++). Both the stack and the heap can be exploited by a buffer overflow attack, but the structure of the stack makes it extremely susceptible.

As its name suggests, the stack is organized as a stack of memory. The stack grows “down” from high memory addresses to lower ones. The current position in the stack is indicated by a variable (the stack pointer) that points to the current top of the stack. As data is added to or removed from the stack, the stack pointer is updated as well.

As shown in the image above, the stack contains several different types of variables. When a function is called by another function, information is pushed onto the stack to provide that function with the data that it needs to execute. This data is pushed onto the stack in the following order:

1. Arguments to the called function (in reverse order)
2. The address of the next instruction after the called function returns
3. Local variables of the called function

Typically, user input to a function will be stored in a local variable, meaning that it will be placed in the memory space directly above the return address on the stack. This is useful to an attacker performing a buffer overflow, since the memory that will be overwritten by an overflowed buffer is the pointer to the next instruction to be executed.

Return-Oriented Programming (ROP)

The fact that an attacker can overwrite the return address of a function on the stack is the basis for return-oriented programming (ROP). In ROP, an attacker attempts to exploit a buffer overflow that causes the vulnerable function to return to an area of the program under the attacker’s control.

This area may be the same buffer overflowed during the attack or some other user-controlled area. If successful, the attacker may be able to convince the application to interpret the provided input as program instructions, allowing the attacker to execute malicious shellcode.

One of the primary challenges of ROP is developing code that does what the attacker wants within a constrained amount of space. For this reason, shellcode commonly tries to call library functions that are already inside the memory space of the process to shorten the necessary code. Some mitigations against ROP focus on making this functionality more difficult for shellcode to locate and execute.

Buffer overflow mitigations

Buffer overflow exploitation can be a serious threat to security since ROP code inserted and executed by the attacker executes with the same privileges as the exploited application. However, multiple means exist for preventing or mitigating buffer overflow attacks.

Stack protections

The primary goal of a buffer overflow exploit is to allow the attacker to run arbitrary code via return-oriented programming. Several different solutions have been implemented to help protect against ROP.

Stack canaries

In order for ROP to be possible on the stack, the attacker needs to be able to rewrite a function return address to point to a region of memory under their control. The concept of a stack canary was invented to help detect and prevent this. 

A stack canary is a value known to the program that is inserted before the return address on the stack. Before a function returns, the value of the canary is checked, and an error is thrown if it is not correct (indicating that a buffer overflow attack occurred).

Data execution prevention

Return-oriented programming relies upon user input intended by the program to be interpreted as data to be interpreted as code instead. This is possible because data and control information are often interwoven on the stack without clear boundaries.

Data Execution Prevention (DEP) marks certain regions of memory as non-executable. This helps protect against buffer overflows since, even if the attacker can modify a return address to point to their shellcode, it won’t be executed by the program. However, DEP can be bypassed by a return-to-libc attack, making address space layout randomization (ALSR) necessary.

Address space layout randomization

Most applications are designed to be object-oriented, with applications making heavy use of shared libraries that they import into their memory space. While the functions in these shared libraries are useful to legitimate code, they’re also useful for ROP.

Address space layout randomization (ASLR) is designed to make it more difficult for an attacker to find the library functions that they need. Instead of importing library functions to set addresses in every application, ASLR randomizes where a particular library will be imported. This makes ROP more difficult, since the attacker needs to find a library in memory before they can use its functions.

Input validation

Buffer overflow attacks are caused when an attacker writes more data to a block of memory than the application allocated for that data. This is possible for a number of reasons, but the most common is the use of unbounded reads that read until a null terminator is found on the input. By using fixed-length reads designed to fit within the allocated buffer space, an application can be made immune to buffer overflow attacks.

Integer overflow checking

Buffer overflows can also be enabled by integer overflow vulnerabilities. This occurs when the length of the value stored in a variable is greater than the variable can hold, causing the variable to drop the most significant bit(s) that do not fit. As a result, a very large input can be interpreted as having a shorter length due to an overflow, causing the allocated buffer to be undersized. 

Checking for integer overflows in input lengths is important for protecting against buffer overflow attacks.

Standard libraries

While C++ allows a developer to manually allocate memory for user input, that doesn’t mean that it is always a good idea to do so. The C++ Standard Template Library (STL) has functions (like strings) that handle memory management correctly behind the scenes. Switching from C-strings to strings is an easy way to mitigate the threat of buffer overflow vulnerabilities.

Conclusion: Buffer overflows for ethical hacking

Buffer overflows are a simple vulnerability that is easily exploited and easily fixed. However even today, software contains exploitable buffer overflow vulnerabilities. In October 2018, a buffer overflow vulnerability was discovered in Whatsapp that allowed exploitation if a user just answered a malicious voice or video call. 

These vulnerabilities are definitely worth checking for when performing an ethical hack and should be corrected in code as quickly as possible.

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

Sources

• Function Calls, Part 3 (Frame Pointer and Local Variables), Codeguru
• Getting around non-executable stack (and fix), SecLists
• CVE-2019-3568 Detail, NIST