Lab3:  Backdoors in OpenSSL

Congratulations!  You’ve just been hired by the Imaginary Security Agency (the ISA) (submit this agency’s name in your submission for extra credit) (REDACTED).  Your first job:  to take one of the most popular cryptographic libraries out there -- OpenSSL -- and insert several vulnerabilities that you can later exploit for your employer’s benefit. Fortunately, the situation for the engagement is such that your inserted vulnerabilities don’t have to be too discreet -- they just have to work.

Here’s the rules of engagement.  You know that a company is going to roll out some new crypto products soon.  Those products are only going to have to interoperate with the company’s own products.  That makes inserting a backdoor / vulnerability so much easier!  Why?  Because that means that the backdoored crypto product does not have to interoperate with someone else’s non-backdoored crypto product -- you can “break” the crypto in pretty significant ways and, assuming the product still works the way it is advertised, no one may ever know.

Each of the products are simple encryption algorithms to encrypt and decrypt plaintext using the OpenSSL libraries.  Your job is to modify the OpenSSL library by using a patch, without modifying any of the products (backdoor_#.c).  For each problem, we’ll apply your modifications to the OpenSSL library, and use your modified library to encrypt some plaintexts.  

Getting Started:

Sign up in Groups by entering your UWNetIDs here:

Once you’ve created your group, you should be able to submit patches and receive files.

We’ve set up a VM image that you can download here:



You don’t have to use the VM, but if you don’t, we won’t respond to any of your questions on the forum, and especially via email.

Once you download the zipped file, extract:

tar xvfz lab3vm.tgz

You can then load the image in Virtualbox or VMware.  On this VM, you should have an openssl directory in your home folder, already pre-compiled to run on your machine.  

Running the Code:

You can use this to test any changes with your own input

cd backdoors/


./backdoor_# “input”

Once you’ve made some changes to the library code and want to test them out, you must re-make your libraries by running ‘make’ (but not make-install) in the openssl directory.  You also need to re-compile the backdoor you’re testing.  We’ve supplied that code in the comments for each backdoor_#.c, it looks something like this if you’re in the backdoors/ directory:

`gcc -I/home/user/openssl/include -L/home/user/openssl/ backdoor_1.c -lcrypto -ldl -o backdoor_1`

We’ll do the previous steps with a wrapper script, which we’ve provided for you in the backdoors/ directory.  However, we will be loading plaintexts and keys from a database you don’t have access to, so you’ll have to create your own input plaintexts and keys.  Feel free to change our code to facilitate your tests, but we will run the exact code you see in the backdoors/ directory when grading your submissions, by running executing ``.

Submitting Patches:

For convenience, we’ve provided an openssl-clean directory which is a copy of the openssl directory.  This matches the directory name on our server, so you must keep it this way as well for your patches to apply correctly.  For starters, run diff on the two directories to make sure that there are no differences.  If there are, you can copy openssl-clean into openssl and retry.

You can create a patch by running from your home directory on the VM:

diff -ruN openssl/ openssl-clean/ > filename.patch

Note: You need to have your changes in openssl, and the original in openssl-clean as that is where our grader will try to make the changes.

If you’re working on the project from your home machine, you can run:

diff -ruN path/to/openssl path/to/openssl-clean patch#.patch

But, the names of the directories must match our server (openssl, openssl-clean).  Furthermore, we cap our patch submissions at 48K, so you need to make sure that you make in openssl, and then copy the pre-made directory into openssl-clean if you do the work on your own machine.  Otherwise your patch files will be too large.

Once you have created your patch, we suggest testing it on your VM to make sure that it applies correctly.

If you’ve already created a group, when you’re ready to submit go to:

and submit your patch for the appropriate problem.

After each patch, you’ll need to roll back to the original version of OpenSSL.  After submitting your patch, you can scrap the openssl directory and copy over openssl-clean. You’ll still need to run make for things to run like new.

Your Tasks:

P1:  Upon receiving your patch, we’ll apply it, run our encryption on plaintext 1, and an unknown plaintext 2.  We’ll return to you plaintext 1, ciphertext 1 and ciphertext 2.  Your job is to patch OpenSSL in a way that allows you to recover plaintext 2.

In performing the encryptions of the two plaintexts, we’ll use the same key and a nonce generated randomly.

Hint: AES_set_encrypt_key calls an assembly code library.  If you look at the Makefiles, you can see which files are compiled into the library.  If you choose to go this route, you can try editing assembly, or changing the makefiles to build different files.

P2:  Upon receiving your patch, we’ll apply it, run our encryption, and return a ciphertext.  Your job is to patch OpenSSL in a way that allows you to recover the corresponding plaintext.  In performing the encryption, we’ll use a key generated by a password based key derivation function and a random password.

P3: Upon receiving your patch, we’ll apply it, run our encryption, and return a ciphertext.  Your job is to patch OpenSSL in a way that allows you to recover the corresponding plaintext.  In performing the encryption, we’ll use a randomly generated public/private key pair, and use the generated public key to encrypt our message.

Submitting to Dropbox: Please submit one text file per group to the course Dropbox in which you describe the method you used to obtain each of these plaintexts and the reason why your method works, and submit each of your patches if they are to be evaluated for partial or extra credit. (One submission per group, and each group member’s name included in that one submission)

Extra credit.  You want to really impress your new employer, so decide to go over and above what’s mandatory for this mission.  You can try to insert other create, hard to detect backdoors.  ((Bonus for hard to detect.  They need to submit both changes to OpenSSL and proof that the library would still “work” with a real system.  They need to submit a writeup explaining how the vulnerability works and why it would be hard to detect.))

Also, if you can make your changes pass make-test, let us know with your submission, and we’ll run make-test.  If you can pass all the tests successfully, you’ll receive extra credit for that backdoor as well.

Due date: Friday December 6

Late policy.  20% late per day.  We will do this by the timestamp of your submission.  If you continue working and submitting on any problem after the deadline, even if you’ve already solved the problem, you will be subject to the late submission deduction.  This includes extra credit.  If you’ve finished working, stop before the deadline and don’t submit anymore, or you will lose points.



will likely be helpful

Base64 decoder:


We’d like some feedback on the lab.  Feel free to submit feedback with your dropbox submissions, or Anonymously here:



1. I don’t think my changes are linking, is there something wrong with ____?

Maybe, but make sure you have re-compiled the backdoor with something like:

gcc -I/home/user/openssl/include -L/home/user/openssl/ backdoor_1.c -lcrypto -ldl -o backdoor_1

If you are using our Makefile, make sure to
make clean

If you’ve done that after each make, ensure that your changes are to the correct function.  For example, AES_set_encrypt_key is defined in three files (from grep):




You can add in a print statement or step along in a debugger to follow the library calls, though stepping in a debugger might be confusing given the OpenSSL program flow.

We suggest using github, grep, and print statements.

2. I’m not getting valid responses from my file submissions.

Make sure that you have created your patch using the `-ruN` option as it tells our system which file to patch.

If you’ve done that, wait a minute.  Ok now check again.

We mark it as processed at the start of processing so that we don’t re-process in the event of a failure.  That places a file there, even if it’s empty or partially formed.

3. How much can I use the forum to discuss this lab?

As always, our goal is for you to learn as much as possible.  Group discussions can help.  Please don’t share your actual solutions on the forum (code or the methods you use).  But please feel free to use the forum to share knowledge about how OpenSSL works.  (We want people to learn OpenSSL, and have some practice with crypto, but we don’t want people to spend too much time getting stuck with odd properties of the OpenSSL codebase.)

For in-person discussions, you can also brainstorm backdooring methods, but please follow the rules specified in the course syllabus about sharing, including the Gilligan’s Island rule and identifying (naming) the people that you brainstorm with in your submission.

Discussing any of the code supplied in the VM is acceptable on the forums.