Ethical hacking: Breaking cryptography (for hackers)

Cryptography is the science of protecting secrets. As a result, it’s designed to make it impossible for an unauthorized party (like a hacker) to gain access to the protected data. While early encryption algorithms had significant flaws and were easily broken, the state of the art in encryption has gotten a lot better.

That being said, cryptography can be broken under the right circumstances. 

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

The first step in breaking cryptography is identifying the cryptographic algorithm in use. This can be accomplished in a number of ways including:

• Manual reverse-engineering
• Keyword searches for algorithm names
• Analysis of an executable’s dependencies
• Ciphertext analysis (e.g., observing block lengths)

With this information in hand, it’s time to start looking for an exploitable vulnerability.

Exploiting cryptographic vulnerabilities

Modern cryptography is secure by design. However, security by design doesn’t necessarily mean perfect security. If cryptography is poorly implemented or misused, vulnerabilities can be created that a hacker can exploit. Many different ways exist to exploit modern cryptosystems.

Key management

Encryption algorithms are designed to ensure that only the sender and recipient of a message are able to access it. This access is protected using secret keys that are known only to the two authorized parties.

However, key management is one of the most common ways that cryptography fails. Key management systems have the classic problem of having to balance usability and security, and usability often wins out. Encryption keys are often based off of or protected by passwords. Since many users commonly use weak or reused passwords, stealing the encryption key is often an effective way of bypassing cryptographic protections.

Stream cipher vulnerabilities

Encryption algorithms come in two main flavors: block and stream ciphers. In general, block ciphers are more intuitive and easier to use, but less efficient (since you need to work in set block sizes). As a result, developers occasionally use stream ciphers.

While stream ciphers can definitely be secure, they also have their shortcomings. One potential issue is the fact that stream ciphers perform encryption at the bit level. If the developer isn’t performing integrity checks, a hacker may not be able to read the protected data, but they can modify it without detection.

Weak/broken encryption algorithms

Another major issue that exists with stream ciphers (and block ciphers as well) is the use of insecure encryption algorithms. RC4 is probably the stream cipher with the highest name recognition, and that name recognition means that it gets used.

However, those developers using it don’t always know how to use it properly. RC4 has a serious cryptographic weakness where the first bits of its output bitstream are strongly correlated to the secret key. If a developer fails to use RC4 (i.e., discarding these bits) and the attacker can guess the output bitstream (i.e., if they can choose the input to the cipher), they may be able to derive the encryption key or at least enough bits to make it vulnerable to a brute-force attack.

Despite its flaws, RC4 is still considered a usable cipher if it is used correctly. Other encryption algorithms (like DES) are known to be vulnerable. Researching encryption algorithm vulnerabilities may allow a hacker to defeat the cryptography.

Custom algorithms/protocols

“Never roll your own crypto” is essentially a law of cryptography. Expert cryptanalysts can spend years developing a cipher, and it can be quickly broken through a vulnerability that they didn’t consider.

Despite this, people still believe that they can create their own algorithms that are better than the standardized ones. If you can get a copy of the software and it has a function, with an excessive number of XORs and bit shifts, it’s probably a custom algorithm. This is definitely worth the effort of reverse-engineering, since it’s almost certainly insecure.

Algorithm misuse

Another common mistake made by developers is misusing algorithms. For example, encryption of traffic between a server and a client should always use asymmetric cryptography. Otherwise, the encryption key is embedded in the client code somewhere. If you think that the code in question is using symmetric encryption for communications, it’s time to go hunting through the executable for that key.

Algorithm misuse also covers a lot of other mistakes. Key reuse can be a significant issue for certain algorithms, so look for a function that generates a shared key between the client and the server. If there isn’t one, the algorithm may be vulnerable to attack. If there is one, it’s possible that a man-in-the-middle attack can allow traffic decryption.

Algorithms can also be misused by failing to properly use certain values. If an encryption algorithm:

• Hashes a password without a unique salt
• Uses a non-random or non-unique initialization vector

… then it’s probably vulnerable to attack. Looking up the specific algorithm and how this affects its security can be the key to breaking it.

Side channels

Side-channel analysis is an attack against cryptography where unintentional data leaks exist because of how the system is implemented. For example, the power consumption of some cryptographic algorithms depends on the secret key in some way.

Trying to measure the power consumption of an algorithm may not be feasible, but execution time is another common side channel. Some algorithms have execution times that are based off of their inputs.

A non-cryptographic example is an authentication system that checks a password character by character and immediately returns when a password is identified as incorrect. This type of system can be defeated by guessing a password one character at a time and observing execution time. Whichever character takes the longest to process is probably the correct one for that bit of the password. If an encryption algorithm has similar timing leaks, it may be vulnerable to attack.

When performing side-channel analysis, it’s important to consider every possible cause for a leak. For example, the Heartbleed vulnerability is caused by memory caching, which creates a timing side channel.

Out of scope

Finally, encryption algorithms are only effective at protecting data while it’s encrypted. Encryption can’t protect data before encryption is performed or after decryption. If important data is encrypted at rest or in transit, it’s often best to try to steal the data when it’s not encrypted by monitoring the memory space of the process performing the encryption and any calls to cryptographic libraries.

Conclusion: Breaking cryptography

In the end, cryptography is only really breakable when it’s done incorrectly. Modern encryption algorithms are designed to make brute-force guessing of the secret key the most effective attack vector and to make that computationally infeasible on modern hardware.

However, encryption algorithms are also extremely sensitive to mistakes in design or implementation. If even a tiny error is made, it may make the algorithm vulnerable to attack.

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

Sources

• Stream Cipher Attack: Bit-Flipping Attack, Total Web Security
• CVE-2015-2808, CVE
• The Heartbleed Bug, Heartbleed.com