Reverse engineering an encryption binary using IDA

A step by step SANS KingleCon2 2019 holiday hack challenge case study 

This article was also published as part of my write-up at the following URL: 
https://medium.com/@walid.daboubi/sans-kringlecon-holiday-hack-challenge-2019-write-up-b47dbb7ce0c1

Context 

The following is the content of the objective 10 question.

The Elfscrow Crypto tool is a vital asset used at Elf University for encrypting SUPER SECRET documents. We can’t send you the source, but we do have debug symbols that you can use.
Recover the plaintext content for this encrypted document. We know that it was encrypted on December 6, 2019, between 7pm and 9pm UTC.
What is the middle line on the cover page? (Hint: it’s five words)
For hints on achieving this objective, please visit the NetWars room and talk with Holly Evergreen.

Git repository

All the used material are available at: https://github.com/slrbl/SANS-KringleCon-Holiday-Hack-Challenge-2019/tree/master/10

Step1

Playing around with the executable 

Let’s try to launch the Elfscrow.exe and see how does it work in term of required input arguments and output.

As we can see, the executable could be used to both encrypt and decrypt files. This interesting because we can consider using it to decrypt the document. We also can notice through the message in the bottom “You can optionally pass — insecure to use unencrypted HTTP. But if youdo that, you’ll be vulnerable to packet sniffers such as Wireshark that could potentially snoop on your traffic to figure out what’s going on!” that to tool to communicates with a remote server.

Let’s try to execute Elfscrow.exe by encrypting a test document which contain “Hello World!” and then decrypting it. In the meanwhile, let’s we can launch Wireshark, apply a HTTP filter and start capturing.

In order to encrypt out test document, we launch the command:

C:\Users\Administrator\Downloads>elfscrow.exe --encrypt --insecure test_document.txt encrypted_test_document.txt

We get the following output:

C:\Users\Administrator\Downloads>elfscrow.exe --encrypt --insecure test_document.txt encrypted_test_document.txt
Welcome to ElfScrow V1.01, the only encryption trusted by Santa!
*** WARNING: This traffic is using insecure HTTP and can be logged with tools such as Wireshark
Our miniature elves are putting together random bits for your secret key!
Seed = 1578901591
Generated an encryption key: 6eec852fbca5a71c (length: 8)
Elfscrowing your key...
Elfscrowing the key to: elfscrow.elfu.org/api/store
Your secret id is bca2ff81-fdd5-4ac3-8aa4-a2d4e2b45154 - Santa Says, don't share that key with anybody!
File successfully encrypted!

So we get the following outputs:

• The encrypted document saved to encrypted_test_document.txt
• A seed: 1578901591
• An encryption key: 6eec852fbca5a71c
• A secret id: bca2ff81-fdd5–4ac3–8aa4-a2d4e2b45154

Let’s have a look on the Wireshark captures:

What we can see in that the key was stored in the server 35.188.222.208 which returned a secret id. We can conclude that the secret id is just database id and doesn’t seem to have big importance in the encryption process itself.

Let’s now try to decrypt encrypted_test_document.txt. We launch the command:

C:\Users\Administrator\Downloads>elfscrow.exe --decrypt --id=bca2ff81-fdd5-4ac3-8aa4-a2d4e2b45154 encrypted_test_document.txt decrypted_test_document.txt

We get the following output in addition to the file decrypted:

C:\Users\Administrator\Downloads>elfscrow.exe --decrypt --id=bca2ff81-fdd5-4ac3-8aa4-a2d4e2b45154 encrypted_test_document.txt decrypted_test_document.txt
Welcome to ElfScrow V1.01, the only encryption trusted by Santa!
Let's see if we can find your key...
Retrieving the key from: /api/retrieve
We found your key!
File successfully decrypted!

Yes as you guessed, the next step is to have a look at the shark 

As we can see, there is nothing special in Wireshark. The tool just retrieved the same key from the server and used it for decryption.

The interesting information we can get from here is that the same key is used for both encryption and decryption.

Step2

Disassembling the executable and understanding how it does the encryption 

Let’s move to more funny stuff. We start by opening IDA, and disassembling Elfscrow.exe

As we can see, we get a list of interesting functions in the functions window. We can try to see what the main function leads us, but nothing really clear.

As we are trying to understand how does the encryption work, let’s try We can then check the do_encrypt function.

We can see that it starts by reading the file to encrypt

It then imports the key, encrypts the input file and finally stores the key

As we are interested to get how the key is generated, we will have a look at generate_key function.

As we can see, generate_key function is calling time function and then giving the the result (stored in EAX) as an input for super_secure_srandfunction.

What we can conclude it that the key generation may be based on current. The make sure about this conclusion we can make a small experience:

• Add a breakpoint at push EAX before calling super_secure_srand

• Check the value returned by time (stored at EAX)
• Modify the value of EAX
• Check if the value returned by generate_key changes if we change the time. This could be done by adding a breakpoint at push EAXjust before calling print_hex (because we are pushing a value that contains a the key to be displayed to the stack, as an argument for the function call. And since we are pushing EAX, it should contain the key)

Truth moment, let run the executable. This could be done by adding the necessary arguments by using Debugger->Process options.

We now can run the executable.

As we can see below It breaks at the first breakpoint, we double click on the RAX in General registers list to get the value currently stored in EAX. It displays 0x000000005E1C3235which is 1578906165,hum it looks like a timestamp. Yes, indeed it’s the current time Monday, January 13, 2020 9:02:45 AM.

What we can do now is the modify the value stored in EAX (the timestamp) with the seed (yes, it’s a timestamp, we can check it by converting it to human readable time) we get when testing the executable in the beginning (1578901591) and check if we get the same key(6eec852fbca5a71c).

By keeping in the same breakpoint, we change the value stored in EAX:

We continue the execution until the next breakpoint we defined. And check the value of the key (stored in EAX before getting pushed to the stack).

As we can see in the previous instruction just before push, we moved a memory address the EAX. Let’s check the value of EAX (the memory address).

Let’s now check the content of that address by clicking on jump in a new window.

And, bingo It contains the key 6eec852fbca5a71c

We now are totally sure that the key is generated based on the timestamp.
As the the Elfscrow.exe uses the same key generated at encryption to decrypt the encrypted file, and as we now that the encryption was done on Dec 12thbetween 7pm and 9pm. We can consider writing a program that given a timestamp produces a key using the same logic in Elfscrow.exe and that trying all the timestamps between Dec 12th7pm and Dec 12th9pm. Before applying this idea we need to understand two things:

• What is the logic Elfscrow.exe uses to generate key based on a timestamp
• How does the decryption work in term of used functions/librarie.

Step3

Understanding how is the function generate_key transforming the seed (timestmap) to a key

When having a look at the code in generate_key function we can see two interesting function, super_secure_srand and super_secure_rand.

super_secure_srand is taking the content of EAX (which is the timestamp) as argument. We concluded this in the previous experiment above.

Let’s have a look inside super_secure_srand

We we can to get a more concrete idea of what it is doing, we set a breakpoint at mov state, ecx and check the value of ECX.

It contains the same seed value 1578901591. We now know that this function set the state variable to variable to the seed value.

Let’s now have a look at super_secure_random function.

The first thing we can notice is that it is using the state variable which contains the seed value. Let’s try to figure out what is this function doing.

As we can see starting from the @ 012B1DC3 it:

• It sets EAX to state (the seed value)
• It multiplies the seed by 343FDh and save the result in EAX
• It adds 269EC3h to EAX
• Saves the value of EAX in state
• Save the value of state in EAX
• Shifts 10h bits of EAX to the right
• Make and ADD operation of EAX and 7FFFh and save the result to EAX

Interesting, we now know how does super_secure_random work. When we get outside the function we can see that it is put inside a loop that repeats 8 (which is the key length) times.

• Update EAX to it’s new value (using super_secure_random)
• Copy the low part (AL) of the value store in EAX to ECX
• Make and ADD operation of ECX and 0FFh and save the result to ECX
• Append CL to a memory address stored in EDX

To get more sense about what’s happening let’s set a break point at the @ 0x012B1E4B and keep watching the content of CL (Low part of ECX)for the 8 iteration.

Well, we know understand that chunks of the key are stored on every iteration in CL to finally get final key 6eec852fbca5a71c which will be stored at [EDX].

Now that we know the algorithm used to generate the key we can easily write a C program to imitate it, something like (it worked for me):

#include <stdio.h>
#include <windows.h>
#include <strsafe.h>

long multiply(long long x, long long y) {
    return  (long)x * y;
}

int main() {

    long timestamp = 1578901591;

    int KEY_LENGTH = 8;

    long seed = timestamp;

    long state = 0;

    char* hex_chunk = "";

    char key[16] = "";

    int len;

    for (int i = 0; i < KEY_LENGTH; i++) {

        hex_chunk = "";

        state = multiply(seed, 0x343FD);

        state = state + 0x269EC3;

        seed = state;

        state = state >> 0x10;

        state = state & 0x7FFF;

        sprintf(hex_chunk, "%x", state);

        len = strlen(hex_chunk);

        const char* last_two = &hex_chunk[len - 2];

        strcat(key, last_two);

    }

    printf("The seed value is %i\n", timestamp);

    printf("The key value is %s\n", key);

}

This program could be fount at: https://github.com/slrbl/SANS-KringleCon-Holiday-Hack-Challenge-2019/blob/master/10/key_gen.c 

After compiling the code, we execute it and we get the following:

We are now able to generate a timestamp based key using the same logic in Elfscrow.exe ;)

Step4

Understand how is the decryption done (from operational/API point of view)

Let’s start by having a look at the content of do_decrypt function:

This function is calling three main functions:

We observe that each of the above functions takes the result of the precedent one as a parameter (with other parameters).

After a small research we conclude that those function belongs to Windows Wincrypt.h. Documentation about the functions could be done found at https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptimportkey.

Step5

Putting all together: Coding time 

Now that we know how to generate a key with the same logic used in Elfscrow, that the key was generated on Dec 12thbetween 7pm and 9pm and how to use Wincryp API, we can finally write program to decrypt the document and get the original PDF. The C program could be found at: https://github.com/slrbl/SANS-KringleCon-Holiday-Hack-Challenge-2019/blob/master/10/decryp.c

Step6

Bingo time  
The git repository contains a precompiled executable of the decryption program we wrote. But in case you do modification, you can compiling using cl (Visual Studio developer tools) command as the following:

After compiling the program, we launch it and keep waiting until a file with the same size as ElfUResearchLabsSuperSledOMaticQuickStartGuideV1.2.pdf.enc appears in the output files. Please not that the result will be saved in ./result folder.

The following is an example of launching the decryption program and the output you get when it succeeds:

...

After a while, we get the decrypted PDF decrypted_dest_1575663650.pdf. According to the timestamp, it was encrypted at Friday, December 6, 2019 8:20:50 PM.
The decrypted document could be found at: https://github.com/slrbl/SANS-KringleCon-Holiday-Hack-Challenge-2019/blob/master/10/decrypted_dest_1575663650.pdf

The document first page is:

Thank you and enjoy!