Security basics with GPG, OpenSSH, OpenSSL and Keybase

UPDATE: for those short on time, read the following Introduction, What are keys and how do they work? and then skip over the sections “Understanding PKI” and “OpenSSL vs OpenSSH” as these just go into more depth on the technical aspect of different encryption concepts. Just skip until What is GPG? (in there are two sub sections about “OpenSSH”, “SSH Agent” and “OpenSSL”, just skip those until you get to the next “GPG” section and continue all the way from there)

Introduction

This post isn’t meant to be “this is how you do security”. I’m not a security expert. I’m not even a security intermediate! When I titled this post “security basics” I wasn’t kidding. If you’re working with applications and/or servers in production then please consult someone better equipped on the subject of security.

Note: although quite a tough read at times, I would highly recommend “Bulletproof SSL and TLS” written by Ivan Ristić

Now the actual purpose of this post was twofold:

  1. Solidify my own understanding of the tools I’ll be covering
  2. Helping others to also understand the purposes of said tools

Security can be confusing. It’s taken me longer than I care to admit to really understand the things I’ll be discussing here (and even then I’ll likely have missed a lot of important nuances), and with that said: what am I planning on covering in this post? Well that would be…

  • What are keys and how do they work?
  • Understanding PKI
  • OpenSSL vs OpenSSH
  • What is GPG
  • Creating your own keys
  • How to encrypt data using GPG, OpenSSL and Keybase

…and a lot more inbetween.

Plaintext and Ciphers

Throughout this post you’ll see me use words like “plaintext” and “cipher”. It’s important to know what these mean before moving on, so let’s clarify this now:

When a file is said to be plaintext it simply means that it’s unencrypted, whereas a cipher is a noun that refers to a plaintext that has been encrypted. That’s it.

What are keys and how do they work?

Imagine you have a plaintext file which contains a password, and you want to share this file with someone else across the internet (let’s say this someone is our friend “Bob”).

You could open an email (or chat program), attach the file and send it to Bob. But this isn’t very safe because you could have some devious person ‘sniffing’ your network traffic, picking up your communication and subsequently stealing the plaintext file containing the password you’d rather they not get access to.

To prevent this devious person from being able to see the password we would need to encrypt the plaintext document into a cipher and to transfer the cipher instead of the plaintext, meaning if anyone was to interrupt your communication then they would get the cipher and it would be unreadable.

In order to encrypt the plaintext into a cipher we need to use a technique that relies on the concept of a “key”. A key is a mathematical algorithm for turning plaintext into seemingly random alphanumeric characters. A key can be used to both encrypt and decrypt plaintext.

Note: the longer the key, the more secure the encryption will be. This is why, when generating keys, you’ll typically be asked to provide the key size (e.g. 128-bit) you want to use for your encryption key

But in order for your recipient (Bob in this case) to know how to decrypt the cipher you’ve sent to them, they also need to know the key you’ve used.

So how do you let the intended recipient (Bob) know the algorithm you’ve used? You can’t just email them and say “I’ve used algorithm X to turn this plaintext document into a cipher”, because the same devious person who originally sniffed your network traffic and grabbed your cipher will also be able to sniff this additional communication and learn the algorithm (i.e. the key) that was used to encrypt the plaintext; allowing them to decrypt your cipher and to retrieve your super secret password.

Public-key cryptography

To resolve the issue of not being able to safely communicate an encryption key, some clever people designed a scheme known as “public-key cryptography”. The principle idea being that you generate two keys:

  1. A public key
  2. A private key

As you can probably already guess, the “public” key is something that is safe to become public (i.e. if some devious person got a hold of your public key then it’s not that much of an issue), while the “private” key is something you should keep hidden and not share with anyone (it’s very important you protect this file).

But how exactly do these two keys help our situation?

Well, the keys are the mathematical inverse of each other; which means you can encrypt data with either the public or private key, and only the alternating key can be used to decrypt the data. So if you encrypted a plaintext using your public key, then the only way you can decrypt the resulting cipher is by using your private key. But imagine you encrypted a file using your public key: as your private key is something only you have access to it means your cipher is safe from everyone! (as long as your private key stays private)

Now that we have a basic understanding of public-key cryptography, you should be able to see how this can be used to keep our ciphers safe from being decrypted by unintended devious type people. So let’s go back to our previous example where we have a plaintext document that we want to encrypt and only share with our friend Bob…

Public-key example

Note: this example is INSECURE and I explain why afterwards

In this example there are two prerequisites:

  1. We’ve generated our own public/private keys
  2. Bob has generated his own public/private keys

With a basic understanding of public-key cryptography, the steps involved appear quite straightforward:

  • We email Bob and ask him for his public key
  • Bob emails us his public key
  • We encrypt our plaintext using Bob’s public key
  • We email Bob the cipher created using his public key
  • Bob receives the cipher and uses his private key to decrypt it
  • Bob now has a copy of the original plaintext

In an ideal world these steps are fine, but we don’t live in an ideal world. Now you may have already noticed the problem with this process, but if not I’ll clarify why this isn’t secure: the devious network sniffer has intercepted your email communication asking Bob for his public key and the devious person sends back his own public key instead. So at this point you get a public key that you think is Bob’s but which actually belongs to the devious person.

Now when you go to send the cipher back to Bob, the devious person sees your communication going across the wire and intercepts it again and grabs the cipher and is able to decrypt it using his private key and subsequently gets access to the plaintext!

Authentication

We arrive at yet another security problem with encrypting data, and although using something like public-key cryptography helps it doesn’t solve the issue of “authentication”. By this we mean: how to do we know the person we’re communicating with is really who they say they are?

You might think for everyone to securely identify themselves they could publish their public keys online. This would mean instead of people having to provide you with their public key via an insecure communication channel, they could point you to a secure location where their public key resides. This is where a service such as https://keybase.io/ comes in (this is still in preview). There are also more traditional services that you can use, such as: keyserver.ubuntu.com, pgp.mit.edu and keyserver.pgp.com

Note: You can access my public key here: keybase.io/integralist

But unless you’re talking (in real-life or over the phone) with the actual person you want to communicate with, then how do you really know who published the key was the person you think it is? Authenticating people is a difficult problem to solve and this is where PKI (Public-key infrastructure) comes in.

Understanding PKI

Public key infrastructure is built on top of Public-key cryptography. The difference is that PKI introduces the concept of “certificates”, and these certificates are used in the software realm much like we would use a passport.

In the real world, the government is a trusted authority (ok so maybe that’s questionable nowadays, but go along with it please…) and they issue you a passport which contains details and information that uniquely identifies you. In the digital world, a certificate does much the same thing.

Now at this point it’s worth pointing out that certificates are designed to identify websites rather than people and so PKI is built on the premise that you are communicating with a domain/web server. They don’t really help us with regards to the problem we had earlier with transferring a cipher securely (I’ll come back to that issue later).

Note: technically certificates are created using the X.509 standard

What PKI can do is help verify the communication between you (e.g. your web browser) and another website is handled securely and is happening with the correct/relevant endpoint. This is useful because if you’re doing some online banking, you want to be sure that communication between you and the bank are happening privately/securely without anyone being able to sniff your information over the wire. But also, you want to be sure you’re communicating with your bank and not some devious endpoint pretending to be your bank but in fact is getting you to type in your account and password details.

The way that PKI manages the ability to authenticate an endpoint (i.e. some website/service you’re communicating with) is through the use of certificates. When you visit a website you’ll use either the http or https protocols. The latter is what signifies a secure connection. When using https, if the website has a valid certificate, then your browser knows that the communication is happening with the right website.

So how can you trust a certificate? Surely someone can set up a website that looks like your bank, then create a certificate and associate it with their website domain? Yes this is possible; but the idea of PKI is that it is built upon a “web of trust”. Let me clarify what that means…

Certificate Authorities (CAs)

Your web browser has a list of organisations it trusts (known as a “CA” or “Certificate Authority”), and these organisations can issue certificates.

If a website uses a certificate that has not been issued (i.e. “signed”) by one of these trusted CAs, then your web browser will display a warning that you probably shouldn’t continue on to the website as it doesn’t appear to be who it says it is (i.e. the website could be who they say they are - your bank - but we can’t really trust them because the certificate they’ve presented to us wasn’t issued by a CA we know of).

Note: certificates are created and then “signed” using an encrypted signature. This is done using the CAs private key. Because the CAs public key is, well… public, it means our browser can use the public key to verify that the certificate it is presented by a website was indeed issued by a CA we trust and wasn’t created by some devious person/organisation instead

So where do these trusted organisations come from? Well as you can imagine, there is a very high cost and detailed process involved with becoming an authorised CA. This is because we have to implicitly trust them to look after our best interests (and only issue certificates to companies/organisations who have proved their true identity through the CAs own rigorous registration process).

Intermediate CAs

Now CAs will sometimes create “intermediate” CAs. These are organisations who can issue certificates on behalf of the original CA (also known as the “Root CA”). If you go to a website that has a certificate, you can inspect the certificate to verify whether it was issued by a root CA or by an intermediate CA. If the certificate came from an intermediate CA, then you can follow the thread back to the root (the web browser typically handles this verification check for you).

One of the reasons this is done is because the root CA is very very important. It has the power to issue certificates and so if the private key ever fell into the wrong hands, then it could be used to generate certificates for all sorts of domains/websites that weren’t who they claimed to be.

In the real world, once a root CA is set-up, the private key is stored offline. For example, the hard drive the private key is stored on is extracted from the computer and stored in a fire safe (even input ports are filled with glue, preventing someone from stealing the drive and trying to extract the data). Serious business this CA stuff!

Certificate Revocation List (CRLs)

Certificates are issued with a validity period (expiration date). Every time your browser interrogates a site’s certificate, it is checking the certs validity period. If the date for the validity period has passed, then the browser will warn you that the certificate is now expired.

At the same time though, if the certificate hasn’t expired, then your browser will consult its Certificate Revocation List to see if the certificate has been revoked. This CRL is downloaded by your browser/operating system on a regular basis and there in lies the problem with CRLs: they’re not real-time results.

Imagine a certificate was issued for the website www.foo.com, but later needed to be revoked (for whatever nefarious reason). In this scenario the CRL is updated to state the website www.foo.com has a revoked certificate and so it cannot be trusted. But because the CRL has to be downloaded in order to see the updated list, the user (you) could end up visiting the website before you had the new CRL and so the certificate would still be seen to be valid.

Because of the lack of real-time validation checking, the Online Certificate Status Protocol (OCSP) has superseded CRLs in that it is (as the name would suggest) an online resource which systems can query at run-time to verify the validity of a certificate.

SSL and TLS

So far we’ve been talking about certificates being the solution to how we can authenticate a server’s identity, and PKI as the overarching process for helping us to secure that communication (using public-key cryptography under the covers).

To help PKI achieve its goals, a cryptographic protocol was designed called SSL (Secure Socket Layers). This protocol was subsequently superseded by a new protocol called TLS (Transport Layer Security). PKI uses these protocols to enable the secure communication.

Note: you might wonder why you don’t hear the phrase TLS used much, and instead see SSL referenced everywhere on the internet when talking about PKI security? This is just an unfortunate case of SSL having become a marketing term that most people can recognise and understand. The majority of the time if someone mentions they have SSL enabled, then what they probably really mean is that they’re using the TLS protocol

SSL handshake (Cipher Suites and Key Exchanges)

In order to secure the communication between the client and the server, PKI uses the stages defined within its protocol to fufil what’s commonly referred to as the “SSL handshake”. This is a set of communicative steps taken between the client (your web browser) and the server.

Remember from earlier we discussed how public-key cryptography works and that with it we can secure the communication channel; but at this point we’re still not sure how that happens without exposing the encryption key (necessary to encrypt our data back and forth across the wire) to any devious people sniffing our network traffic.

As we’ll see in a moment, one of the steps in the SSL handshake is called the “key exchange”; this exchange between the client/server is for the encryption key, and is done using a public-key cryptography algorithm. The most popular choice (at the time of writing) is the RSA algorithm, which uses the server’s public key (provided in the certificate the server sends to the client) to encrypt the key before sending it to the server.

Note: if you’re using the Diffie–Hellman key exchange algorithm you’ll find a great visual explanation of the process which uses the analogy of “mixing colours” to indicate the maths behind the equation (e.g. easy to calculate in one direction, but very difficult to reverse; much like mixing two colours together is easy, but unmixing would be quite arduous). There are also performance penalties associated with some more advanced key exchange algorithms that you need to take into consideration

In order for the SSL handshake to proceed successfully, the client needs to provide the server with some preliminary options; one being a “cipher suite”. A cipher suite has a structure that looks something like the following:

TLS_RSA_WITH_AES_128_CBC_SHA

This might just look like a jumble of acronyms, so let’s break down what this means:

TLS: the protocol
RSA: the key exchange algorithm
AES: the encryption algorithm
128: the encryption strength
CBC: the encryption mode
SHA: the hash function for a digital signature

In the above example, we use RSA which is interpreted as both the key exchange algorithm AND the authentication mechanism. But with other cipher suites you’ll see two separate values (e.g. DHE_RSA where DHE is the key exchange and RSA is the authentication mechanism).

The client supports various different cipher suites and so it’ll send all of the different variations it is happy to handle, while the server’s job is to find the most secure match and respond to confirm the cipher suite it has selected.

Note: cipher suites are just one (of many) areas of communication open to a MITM (man-in-the-middle) attack. For example, a devious network sniffer intercepts your initial insecure communication with a server and removes all the cipher suites leaving only the weakest one. The server has no option but to select the one and only cipher suite left, meaning the attacker has an easier time brute forcing through the weaker encryption methods

One other item we’ll want to be aware of is what’s called a MAC (Message Authentication Code). The MAC is a way of ensuring authentication and integrity by combining an agreed key and a hashing cipher to create a signature for some content. If we send some data we’ll also send a MAC with it and because both sides have the key/cipher information we can ensure the message content hasn’t been tampered with. We’ll see in just a moment that one of our handshake steps will be for the client/server to verify each other using a MAC.

Let’s take a quick look at the SSL handshake (this isn’t exhaustive, and I’ve left out lots of steps for brevity):

  • Client sends cipher suite, a random number, a protocol version and list of compression methods
  • Server sends back a selected cipher suite (most secure match), a random number, the protocol version it supports and a selected compression method
  • Client requests certificate for identification **
  • Server sends its certificate
  • Client sends a “premaster secret” encrypted using the key exchange algorithm defined in the selected cipher suite
  • Server decrypts the premaster secret
  • Both client/server can generate a “master secret” using both sets of random numbers previously sent to each other
  • Client/Server uses the master secret to derive the encryption keys used to encrypt all future communication
  • Client sends a MAC of the communication so far
  • Server creates a MAC and compares with Client MAC (if the same then Server switches to encryption)
  • Server tells the client it’s ready for secure messages
  • Client sends secure mesage(s)

Example messages

In the previous sub section I briefly ran through the different steps the client and server take in order to communicate securely with each other. But I’d like to add onto that some examples of these messages.

Note: these examples are copied verbatim from the excellent book “Bulletproof SSL and TLS” written by Ivan Ristić

Here is the first example, this is the client opening communication with the server:

Handshake protocol: ClientHello
    Version: TLS 1.2
    Random
        Client time: May 22, 2030 02:43:46 GMT
        Random bytes: b76b0e61829557eb4c611adfd2d36eb232dc1332fe29802e321ee871
    Session ID: (empty)
    Cipher Suites
        Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”
        Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        Suite: TLS_RSA_WITH_AES_128_GCM_SHA256
        Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        Suite: TLS_RSA_WITH_AES_128_CBC_SHA
        Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA
        Suite: TLS_RSA_WITH_RC4_128_SHA
    Compression methods
        Method: null
    Extensions
        Extension: server_name
            Hostname: www.feistyduck.com
        Extension: renegotiation_info
        Extension: elliptic_curves
            Named curve: secp256r1
            Named curve: secp384r1
        Extension: signature_algorithms
            Algorithm: sha1/rsa
            Algorithm: sha256/rsa
            Algorithm: sha1/ecdsa
            Algorithm: sha256/ecdsa”

As you can see, all the ingredients are there as we described earlier; the cipher suite being the most important to take note of at this time. Let’s move on and see what the server’s response would typically look like:

Handshake protocol: ServerHello
    Version: TLS 1.2
    Random
        Server time: Mar 10, 2059 02:35:57 GMT”
        Random bytes: 8469b09b480c1978182ce1b59290487609f41132312ca22aacaf5012
    Session ID: 4cae75c91cf5adf55f93c9fb5dd36d19903b1182029af3d527b7a42ef1c32c80
    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    Compression method: null
    Extensions
        Extension: server_name
        Extension: renegotiation_info”

Here we can see the server has sent back its random data (used to construct the premaster secret) and also we can see which cipher suite it has selected to be used.

Verify SSL

Sometimes you might need to debug an issue with your SSL connection. In order to do that you’ll want to utilise the OpenSSL utility command s_client. This will allow you to open a connection to your host using the SSL/TLS protocol of your choice and control the various different configuration settins.

A basic example would be as follows:

openssl s_client -connect google.com:443 -showcerts

You’ll see we’re connecting to Google (which is secured using SSL/TLS) and we also specify the -showcerts flag, which allows the response to display all certificates provided within the chain.

The response looks something like the following:

CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority G2
-----BEGIN CERTIFICATE-----
MIIHjTCCBnWgAwIBAgIIGo+tp3jIDvEwDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTUxMDE1MTY0MTE2WhcNMTYwMTEzMDAwMDAw
WjBmMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEVMBMGA1UEAwwMKi5n
b29nbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApCZxkmtf
Au04O0RjThQnFAzC27w9VlfAoQrgqPtarJIvQDw4G8NZEUSWIlPr0nGr5z3CicxY
ddO2+U8vz/aEltcIUDqNNsdordbLLl7zBS2zkDnSZEmJcM5wYM8biJ3FSU3WzRmN
o8px1LCj49dpQButCJ7FCDIU3eaxZRYZaEiO2UCR7BT8wdyE9O49YKcDUycTgaNu
pM5oTFPqqj+3mDLZsB8BjIssboE8dutdU0TTY4SJ3EPFsSeVTOFu7WY+s628hwMw
kw5sy9viYUZf11RL0zleKarvdQ+oE6gK59ilknrrRXONpepKX6cBGwF/cF+Xl4y4
tjvbvWfNETvmPQIDAQABo4IEWjCCBFYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMIIDJgYDVR0RBIIDHTCCAxmCDCouZ29vZ2xlLmNvbYINKi5hbmRyb2lk
LmNvbYIWKi5hcHBlbmdpbmUuZ29vZ2xlLmNvbYISKi5jbG91ZC5nb29nbGUuY29t
ghYqLmdvb2dsZS1hbmFseXRpY3MuY29tggsqLmdvb2dsZS5jYYILKi5nb29nbGUu
Y2yCDiouZ29vZ2xlLmNvLmlugg4qLmdvb2dsZS5jby5qcIIOKi5nb29nbGUuY28u
dWuCDyouZ29vZ2xlLmNvbS5hcoIPKi5nb29nbGUuY29tLmF1gg8qLmdvb2dsZS5j
b20uYnKCDyouZ29vZ2xlLmNvbS5jb4IPKi5nb29nbGUuY29tLm14gg8qLmdvb2ds
ZS5jb20udHKCDyouZ29vZ2xlLmNvbS52boILKi5nb29nbGUuZGWCCyouZ29vZ2xl
LmVzggsqLmdvb2dsZS5mcoILKi5nb29nbGUuaHWCCyouZ29vZ2xlLml0ggsqLmdv
b2dsZS5ubIILKi5nb29nbGUucGyCCyouZ29vZ2xlLnB0ghIqLmdvb2dsZWFkYXBp
cy5jb22CDyouZ29vZ2xlYXBpcy5jboIUKi5nb29nbGVjb21tZXJjZS5jb22CESou
Z29vZ2xldmlkZW8uY29tggwqLmdzdGF0aWMuY26CDSouZ3N0YXRpYy5jb22CCiou
Z3Z0MS5jb22CCiouZ3Z0Mi5jb22CFCoubWV0cmljLmdzdGF0aWMuY29tggwqLnVy
Y2hpbi5jb22CECoudXJsLmdvb2dsZS5jb22CFioueW91dHViZS1ub2Nvb2tpZS5j
b22CDSoueW91dHViZS5jb22CFioueW91dHViZWVkdWNhdGlvbi5jb22CCyoueXRp
bWcuY29tggthbmRyb2lkLmNvbYIEZy5jb4IGZ29vLmdsghRnb29nbGUtYW5hbHl0
aWNzLmNvbYIKZ29vZ2xlLmNvbYISZ29vZ2xlY29tbWVyY2UuY29tggp1cmNoaW4u
Y29tggh5b3V0dS5iZYILeW91dHViZS5jb22CFHlvdXR1YmVlZHVjYXRpb24uY29t
MGgGCCsGAQUFBwEBBFwwWjArBggrBgEFBQcwAoYfaHR0cDovL3BraS5nb29nbGUu
Y29tL0dJQUcyLmNydDArBggrBgEFBQcwAYYfaHR0cDovL2NsaWVudHMxLmdvb2ds
ZS5jb20vb2NzcDAdBgNVHQ4EFgQUpypq8xCS2PlWTuyHYX7xgHb9UgwwDAYDVR0T
AQH/BAIwADAfBgNVHSMEGDAWgBRK3QYWG7z2aLV29YG2u2IaulqBLzAhBgNVHSAE
GjAYMAwGCisGAQQB1nkCBQEwCAYGZ4EMAQICMDAGA1UdHwQpMCcwJaAjoCGGH2h0
dHA6Ly9wa2kuZ29vZ2xlLmNvbS9HSUFHMi5jcmwwDQYJKoZIhvcNAQELBQADggEB
AFRzPPSVNMFRDoePYeOs7awHmRpp79v7loOBMO7ctrTjpfhVjPsQmvzDlfEken2q
HMs/r76I0s7E4hk6v2ZNFiDjSnnHvjIGG1PydfEVovL/FqsX3kUaWqqCeymGcwRC
ld2F6XqgKkPRf00DjbnFmDKGpRyEMQd91HoutzSMSypPqFCNSpVGuPFP32D1+/Vh
oukY3gGUlNGukoJ2e/VNnRrU7NBvyGkW83CGPMR2iwwa0/7aVGs1iy4aypaBo+0q
QxFmxwn1AWJYyhFbsdfrhf3ux8KEQGDEIHOSa9+SjL/GYQENXGfkVIj/OG1KV0CB
K8pC8t/AFLMrBFjS1gbkE7U=
-----END CERTIFICATE-----
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
-----BEGIN CERTIFICATE-----
MIID8DCCAtigAwIBAgIDAjqDMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTMwNDA1MTUxNTU2WhcNMTYxMjMxMjM1OTU5WjBJMQswCQYDVQQG
EwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVy
bmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NP
VaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtv
h8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE
ahqyzFPdFUuLH8gZYR/Nnag+YyuENWllhMgZxUYi+FOVvuOAShDGKuy6lyARxzmZ
EASg8GF6lSWMTlJ14rbtCMoU/M4iarNOz0YDl5cDfsCx3nuvRTPPuj5xt970JSXC
DTWJnZ37DhF5iR43xa+OcmkCAwEAAaOB5zCB5DAfBgNVHSMEGDAWgBTAephojYn7
qwVkDBF9qn1luMrMTjAdBgNVHQ4EFgQUSt0GFhu89mi1dvWBtrtiGrpagS8wDgYD
VR0PAQH/BAQDAgEGMC4GCCsGAQUFBwEBBCIwIDAeBggrBgEFBQcwAYYSaHR0cDov
L2cuc3ltY2QuY29tMBIGA1UdEwEB/wQIMAYBAf8CAQAwNQYDVR0fBC4wLDAqoCig
JoYkaHR0cDovL2cuc3ltY2IuY29tL2NybHMvZ3RnbG9iYWwuY3JsMBcGA1UdIAQQ
MA4wDAYKKwYBBAHWeQIFATANBgkqhkiG9w0BAQsFAAOCAQEAqvqpIM1qZ4PtXtR+
3h3Ef+AlBgDFJPupyC1tft6dgmUsgWM0Zj7pUsIItMsv91+ZOmqcUHqFBYx90SpI
hNMJbHzCzTWf84LuUt5oX+QAihcglvcpjZpNy6jehsgNb1aHA30DP9z6eX0hGfnI
Oi9RdozHQZJxjyXON/hKTAAj78Q1EK7gI4BzfE00LshukNYQHpmEcxpw8u1VDu4X
Bupn7jLrLN1nBz/2i8Jw3lsA5rsb0zYaImxssDVCbJAJPZPpZAkiDoUGn8JzIdPm
X4DkjYUiOnMDsWCOrmji9D6X52ASCWg23jrW4kOVWzeBkoEfu43XrVJkFleW2V40
fsg12A==
-----END CERTIFICATE-----
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
-----BEGIN CERTIFICATE-----
MIIDfTCCAuagAwIBAgIDErvmMA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDIwNTIxMDQwMDAwWhcNMTgwODIxMDQwMDAw
WjBCMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UE
AxMSR2VvVHJ1c3QgR2xvYmFsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA2swYYzD99BcjGlZ+W988bDjkcbd4kdS8odhM+KhDtgPpTSEHCIjaWC9m
OSm9BXiLnTjoBbdqfnGk5sRgprDvgOSJKA+eJdbtg/OtppHHmMlCGDUUna2YRpIu
T8rxh0PBFpVXLVDviS2Aelet8u5fa9IAjbkU+BQVNdnARqN7csiRv8lVK83Qlz6c
JmTM386DGXHKTubU1XupGc1V3sjs0l44U+VcT4wt/lAjNvxm5suOpDkZALeVAjmR
Cw7+OC7RHQWa9k0+bw8HHa8sHo9gOeL6NlMTOdReJivbPagUvTLrGAMoUgRx5asz
PeE4uwc2hGKceeoWMPRfwCvocWvk+QIDAQABo4HwMIHtMB8GA1UdIwQYMBaAFEjm
aPkr0rKV10fYIyAQTzOYkJ/UMB0GA1UdDgQWBBTAephojYn7qwVkDBF9qn1luMrM
TjAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjA6BgNVHR8EMzAxMC+g
LaArhilodHRwOi8vY3JsLmdlb3RydXN0LmNvbS9jcmxzL3NlY3VyZWNhLmNybDBO
BgNVHSAERzBFMEMGBFUdIAAwOzA5BggrBgEFBQcCARYtaHR0cHM6Ly93d3cuZ2Vv
dHJ1c3QuY29tL3Jlc291cmNlcy9yZXBvc2l0b3J5MA0GCSqGSIb3DQEBBQUAA4GB
AHbhEm5OSxYShjAGsoEIz/AIx8dxfmbuwu3UOx//8PDITtZDOLC5MH0Y0FWDomrL
NhGc6Ehmo21/uBPUR/6LWlxz/K7ZGzIZOKuXNBSqltLroxwUCEm2u+WR74M26x1W
b8ravHNjkOR/ez4iyz0H7V84dJzjA1BOoa+Y7mHyhD8S
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
---
No client certificate CA names sent
---
SSL handshake has read 4021 bytes and written 456 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: C9BC3D21CD96713FA4EA7C21D60FF2F5DDBC6C2D2C68543652E9790C7A98DC58
    Session-ID-ctx: 
    Master-Key: AE3A9EBD53ED49E4FDAABBCF6EA3B4096D9248127BC056CFB1F8981F2A3AB4E8779DC8195241A9A3A25DB03EDAED6077
    Key-Arg   : None
    Start Time: 1445868754
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

What might not be clear at this point is you’re still sitting in an interactive mode within the shell and so you can issue additional requests like so:

HEAD / HTTP/1.0
Host: www.google.com

Note: remember to press <Enter> twice to send the request

The above sends a request for just the headers for the specified host, and so the response looks something like the following:

HTTP/1.0 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: https://www.google.co.uk/?gfe_rd=cr&ei=wzUuVtGjLLDj8wek176wCQ
Content-Length: 262
Date: Mon, 26 Oct 2015 14:16:35 GMT
Server: GFE/2.0
Alternate-Protocol: 443:quic,p=1
Alt-Svc: quic="www.google.com:443"; p="1"; ma=600,quic=":443"; p="1"; ma=600

The following is another example (a non-working example unfortunately), but this time the service we’re querying is utilising self-signed certificates, and so we’re required to provide the CA as well as our own client certificate for authentication:

openssl s_client \
  -connect my-fake-service.com:443 \
  -CAfile /Users/Integralist/.pki/custom-ca.pem \
  -cert /Users/Integralist/.pki/Certificate.pem \
  -key /Users/Integralist/.pki/Certificate.key \
  -showcerts

The last example I want to show you is where we try and verify if a particular cipher is disabled (in this case the insecure RC4 cipher):

openssl s_client -connect blog.mozilla.org:443 -showcerts -cipher RC4-SHA

Mozilla released a blog post recently that stated they’ve discontinued support for this particular cipher. So the above command will return the following output, which indicates a SSL handshake failure:

57533:
error:
14077410:
SSL routines:
SSL23_GET_SERVER_HELLO:
sslv3 alert handshake failure:
/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59/src/ssl/s23_clnt.c:
593:

Certificate signature

One way the client and server can authenticate each other’s identities is via the MAC they send during the SSL handshake. But the client first verifies the server’s identity by checking the certificate provided by the server (also done at the very beginning of the SSL handshake) has not only been signed by a CA we trust, but has also not been modified at any point along the way from the server to the client.

The client can do this by inspecting the signature on the certificate. Earlier when we discussed how CAs generate certificates, we mentioned that the CA will “sign” the certificate. The signature is result of applying a hash function to the contents of the certificate itself and then encrypting that hash value using the CAs private key.

The reason this technique works is because if a devious network sniffer tried to replace the certificate with their own modified version, then the client would be able to tell; and the reason the client can tell the certificate has been tampered with is because the client can verify the certificate by converting it into a hash using the same hash function as the CA used when creating its signature (details of the hashing algorithm are part of the certificate). The client can then decrypt the CAs signature (using the CAs public key) which results in the hash value. The client then compares the two hashes (the one it extracted from the signature, and the one it generated itself from the certificate content) to make sure they match. If they don’t match, then we know the certificate has been modified at some point and cannot be trusted.

PKI Conclusion

So there you have it, that’s pretty much how PKI (and subsequently SSL/TLS) works; although presented in a stripped down way to make this post even remotely bearable to any sane person.

Let’s move on!

OpenSSL vs OpenSSH

If you want to generate your own keys and certificates, which will enable you to connect and transmit data more securely across the internet; then you’re going to need to install the OpenSSL command line toolkit. OpenSSL is a library designed to implement the SSL/TLS protocols

Note: the openssl command is a wrapper around the OpenSSL library

What you might not be aware of though is how large a suite of cryptographic tools OpenSSL actually provides. Later on in this post we’ll demonstrate a tiny selection of these tools in order to create our own keys and encrypt some data; but for now we’ll focus more on the differences between OpenSSL and OpenSSH.

So where OpenSSL is designed to provide a method for securing web based communication; OpenSSH on the other hand provides secure and encrypted tunneling capabilities. It is typically used to enable secure shell connections from your machine to external servers.

OpenSSH has a different transport protocol compared to OpenSSL. Although OpenSSH does actually utilise OpenSSL for its cryptographic operations, such as key pair generation. So as long as you’re using the same algorithms for generating keys you’ll find no difference between OpenSSH and OpenSSL’s level of security in that sense (although there is a larger attack vector with regards to OpenSSH so you could argue it’s potentially more open to security concerns as attackers have more options available to them).

Note: the suite of command line tools people typically associate with OpenSSH are actually commands designed around the OpenSSH protocol standard (i.e. there isn’t an actual openssh command; and as such, OpenSSH is just the ‘protocol’ and not the command line tools themselves)

OpenSSH utilities

There are a few different tools available within OpenSSH and although (later on in the section where we learn how to generate our own keys) we’ll be using the command line tools, we’ll primarily be focusing on the ssh-keygen command along with demonstrating how I use ssh-agent and ssh-add to make working with OpenSSH a little easier.

Below are a list of tools that are built upon the OpenSSH protocol:

  • ssh-keygen
  • ssh-agent
  • ssh-add
  • ssh-keysign
  • ssh-keyscan
  • sftp-server
  • sshd

What is GPG?

GPG is a tool which provides encryption and signing capabilities. Its full name is “GNU Privacy Guard”.

GPG supports both symmetrical and asymmetrical encryption techniques along with an optional digital signing of your encrypted content to ensure integrity.

In the next section “Creating your own keys” I’ll demonstrate how to actually use GPG.

GPG vs PGP

You may have also heard of PGP and wondered what the differences are between that and GPG: PGP is the protocol standard (defined under the name “Open PGP”) which GPG implements. So PGP isn’t a tool itself, but merely a specification for other tools (such as GPG) to build upon.

Creating your own keys

OK, up until this point we’ve only been talking in a theorectical sense. Time to see some pratical use cases by demonstrating how to generate your own keys using the three different toolkits we’ve described up until this point (OpenSSH, OpenSSL and GPG).

I’m going to quickly run through each utility (OpenSSH, OpenSSL and GPG) and explain how you can create your own keys for each of them. I wont go into great detail the flags/settings used in each example command as that is what the man command is for (i.e. I’ll leave investigation of these settings as an exercise for the reader).

Also, generating keys is one thing. But for the OpenSSL and GPG utilities, it’s not until we need to actually encrypt some data (see the next section “How to encrypt data using GPG, OpenSSL and Keybase”) that these keys can become useful (OpenSSH is another beast altogether).

So let’s begin…

OpenSSH

In the following example we’re generating a new set of keys (public and private) using the RSA type and using 4096 bits for the key length. This is considered quite a secure set-up (anything less than 2048 bits is easily crackable in todays digital age):

ssh-keygen -t rsa -b 4096 -C "your.email@service.com"

Running this command you’ll be asked to provide a name for the keys and an (optional) password. Once this is done you’ll find two files in the current directory (imagine we named the key foo_rsa when prompted):

  1. foo_rsa: contains your private key
  2. foo_rsa.pub: contains your public key

Note: you can change the passphrase associated with your private key by running ssh-keygen -p

Now we have these keys, we can provide our public key to an external service such as GitHub or have them installed on a remote server. Either way this will allow us to connect our shell securely to these remote services/servers.

In the case of connecting to a remote server, you would have your devops or operations people add your public key into a ~/.ssh/authorized_keys file (or you could do it yourself: cat foo_rsa.pub | ssh user@123.45.56.78 "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"). Once your public key is added you’ll be able to securely connect to the server without requiring your password because your private key will be used to authenticate your access.

Note: convention is for SSH keys to be placed inside a ~/.ssh folder

One final change that can be made on your remote server (again, this could be handled by your devops or operations team) is to restrict logins to your server to only happen via SSH keys. To do this you need to log into the server and change the file /etc/ssh/sshd_config by locating the line that specifies PermitRootLogin and changing its value like so: PermitRootLogin without-password. From there you would run reload ssh for the changes to take immediate effect.

Note: to find the fingerprint of your SSH key use:
ssh-keygen -E md5 -lf ~/.ssh/your_rsa.pub
The -E md5 provides output as shown by GitHub/GitLab

SSH Agent

Most operating systems have ssh-agent available. If you’ve got ssh-keygen installed, then chances are you’ll have the agent and other OpenSSH tools as well. The agent is used to store private keys used for public key authentication. It makes using SSH easier (a trade-off of security for convenience) by allowing you to specify your private key password once.

When I’m setting up my SSH keys for using GitHub I’ll typically run the following commands:

cd ~/.ssh
ssh-keygen -t rsa -b 4096 -C "my.email@domain.com" # saved as github_rsa
eval "$(ssh-agent -s)"
ssh-add -K ~/.ssh/github_rsa
pbcopy < ~/.ssh/github_rsa.pub
ssh -T git@github.com

So what’s happening here is:

  • I move into the relevant directory where my SSH keys are located
  • Generate my SSH keys (saved as github_rsa)
  • Start the SSH Agent
  • Use ssh-add to add my private key to the agent
  • Copy my public key (and manually paste it into the GitHub GUI)
  • Verify the setup

Note: I also use the -K flag with ssh-add as that’s specific to Mac OS X

OpenSSL

Like with the OpenSSH example in the previous sub section, here we’ll be generating a new set of keys (public and private) using the RSA type and using 4096 bits for the key length. The difference is that you have to generate the private key first and then extract the public key from it:

openssl genrsa -out private_key.pem 4096
openssl rsa -pubout -in private_key.pem -out public_key.pem

You can also print out some additional details contained inside your pem file by using the -text flag:

openssl rsa -text -in private_key.pem

GPG

Note: here is a great and detailed article on how to make the most secure key pair process possible

Generating a key pair with GPG is a little bit more involved as you have some prompts you need to step through. The command to begin with is:

gpg --gen-key

This will present the following information:

Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection?

You’ll then be asked the key length (I entered 4096 to match other keys I’ve created):

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)

After this you’ll be asked to provide an expiration date (I picked 1 year):

Requested keysize is 4096 bits       
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)

Finally, you’ll need to input some personal details. I wont add those here, I’ll just let you fill those in as needed. It’s also worth being aware that GPG will use your system’s entropy to aid its random number generation, so for a brief moment you’ll see it request that you move your cursor around a bit to help build up the entrophy output.

Note: you could also provide all these details via an ‘input’ file (useful if you find yourself generating lots of key pairs), but that’s a bit outside the scope of what we want to focus on here. If you’re interested, you can find the details here

Now before we go any further, it’s important that at this point you now generate a revocation certificate:

gpg --gen-revoke my.email@domain.com

You’ll want to put this certificate into a file and store it somewhere very safe in case you need to ever revoke your key pair. Preferably you print it out on paper and put it in a fireproof safe.

If you ever need to, you can then import the certificate into your keyring and that will immediately revoke your key pair:

gpg --import revocation.cert

You can then also push up your key identifier to a key server to force it to recognise the key has been revoked:

gpg --keyserver pgp.mit.edu --send-keys 123456

Once your key pair has been generated you’ll probably be confused as to where they’re stored? If you were to ls the current directory they wont be there. Instead you’ll need to use the following command to see what public keys you’ve generated:

gpg --list-keys

For me this displays the following output:

/Users/M/.gnupg/pubring.gpg
---------------------------
pub   4096R/056C9716 2015-08-14 [expires: 2016-08-13]
uid                  Mark McDonnell (Hi) <my.email@domain.com>
sub   4096R/A1F3D5B6 2015-08-14 [expires: 2016-08-13]

Note: there is an equivalent command for viewing your private keys gpg --list-secret-keys

You can see there is a file pubring.gpg that appears to contain the details of all the keys I’ve created, and interestingly the file itself is protected; so if I try something like cat ~/.gnupg/pubring.gpg it’ll spew out encrypted cipher text at me.

If you want to see your public key, then use the following command (notice I’ve specified the same name as seen from the above output from --list-keys; if I tried to specify something else then it wouldn’t find any key to export):

gpg --export --armor "Mark McDonnell" > gpg_public.key

Note: --armor creates ASCII armored output (a text format) instead of raw bytes (binary format)

To get the private key, you’ll use a slightly different flag, but effectively it’s the same thing:

gpg --export-secret-key --armor "Mark McDonnell" > gpg_private.key

If you have multiple keys under the same name then you’ll find that it’ll typically export the key for the first name it finds. To work around this you can specify the email address instead:

gpg --export --armor my.email@gmail.com

Note: by default it prints to stdout

In the following sections we’ll see more of how to use GPG, but it’s worth mentioning now that all the settings you use in the command line tool can be set as defaults in a dotfile: ~/.gnu/gpg.conf

For example, if on the command line you used --default-key, then you could set the following within your gpg.conf:

default-key 62DBDF62           # Personal
default-recipient some-user-id # In case you happen to only ever communicate with one person

Multiple Keys?

Here’s a quick question that people seem to ask a lot:

I heard some people have multiple key pairs, one for ‘signing’ and one for ‘encryption’. Why is that?

The answer is that you’ll want to rotate your encryption key pair on a regular basis. Ideally you wouldn’t have a key pair that never expires or doesn’t expire for a very long time because if it becomes compromised (and you’re unable to revoke) then you’ve got serious problems.

Also the company you work for might want to control your encryption key so that once you leave the organisation they can decrypt stuff you’ve worked on previously. But if they have your key then you’d need to create a new one for your personal interactions and means you couldn’t build up a secure and well established identity outside of the company.

In all these cases, it can be easier to just have a separate ‘signing’ key pair that lasts for a long time. This ultimately becomes your identity that people trust. You then have separate keys for encrypting and you can digitally sign the encrypted content.

Interestingly by default GPG creates a signing key and an encryption key. You can see this by running the following command:

gpg --list-sigs

I recommend reading https://alexcabal.com/creating-the-perfect-gpg-keypair/ to learn how to create your own sub signing keys and improve the security of your key process/setup.

How to encrypt data using GPG, OpenSSL and Keybase

Outside of PKI and SSL/TLS, the most common task people are interested in is encrypting specific files that contain sensitive information that they would prefer to be protected either from everyone or allow only a select few people to be able to access. The interface for encrypting data is different for each tool used and so we’ll be looking at those we’ve discussed so far: GPG and OpenSSL

GPG encryption

GPG offers two forms of encryption: asymmetrical and symmetrical…

Asymmetrical encryption

With GPG you’ll need the recipients public key in order to encrypt files. So once you have the recipients public key you’ll need to import it into GPG so you can reference it. To do that you’ll use the following command:

gpg --import public.key

Note: to delete their public key afterwards, run gpg --delete-key "User Name"

Before we continue, let’s just consider a real-world scenario:

Imagine at this point you’re not entirely sure if the public key you’ve been given over the internet is actually from who you were expecting it from (let’s call them “Bob”). Maybe there are some nefarious government types intercepting your communication and the public key you received is theirs and not Bob’s.

You would need to call or find some form of secure communication channel with Bob (or someone who you trust to know bob) to verify the public key really does belong to Bob. But what do you check any way? You could read out the contents of the public key (e.g. cat bobs-public.key) but this would be pain stakingly tedious.

Another way would be to get Bob to give you the shortened ‘fingerprint’ which is derived from his public key. You do that by running the following command and asking Bob if it matches what he’s seeing:

gpg --fingerprint bob@example.org

This will return something like:

pub   4096R/DD86E7F5 2016-05-24 [expires: 2017-05-24]
      Key fingerprint = FDFB E9B5 24BA 6972 A3AA  44B9 A1B1 7E6F DD86 E7F5
uid                  Mark McDonnell (Personal) <mark.mcdx@gmail.com>
sub   4096R/A3FDEBC6 2016-05-24 [expires: 2017-05-24]

Notice the Key fingerprint section FDFB E9B5 24BA 6972 A3AA 44B9 A1B1 7E6F DD86 E7F5. That’s what I would read out to someone if they called me to say they wanted to verify the public key they had really did belong to me.

Now, once you have the public key of your recipient you can encrypt a file using it, like so:

gpg --encrypt -u "Sender User Name" -r "Receiver User Name" somefile

Note: it can sometimes be better to use the pub identifier number (especially when you have multiple keys with the same email). So if your pub id is 1234A/BC56D7E5 then you’d use -u BC56D7E5

To decrypt a GPG encrypted file, the person sending you the encrypted file would have used your public key to encrypt the data. So the following command will locate your private key automatically (if you have multiple secret keys it’ll ask for the password):

gpg -d some_encrypted_file.gpg

In the above example, it’ll display the contents of the file in stdout; so you’ll need to redirect it to a file or alternatively use:

gpg -o output_file -d some_encrypted_file.gpg

Symmetrical encryption

If you don’t want to encrypt a file using your own key pair, you can use standard symmetrical encryption:

gpg --symmetric secrets.txt

You’ll be asked to enter a passphrase to complete the encryption. Then if you want to decrypt the file you simply run:

gpg secrets.txt.gpg

It’ll ask for the passphrase and then it’ll extract the file to the current directory.

You can also modify the default cipher encryption algorithm, then use the --cipher-algo flag:

gpg --verbose --cipher-algo AES256 --symmetric secrets.txt

Note: use --verbose without --cipher-algo to see GPG’s default algorithm

To see a list of available cipher algorithms, then execute the following command and look for the section Cipher:

gpg --version

So for me this outputs:

gpg (GnuPG) 1.4.20
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Key Signing

GPG let’s you “sign” a key, and which tells GPG you trust this key you have been provided (i.e. you have verified it belongs to who you think it should belong to). To sign a key you’ve imported, simply run the following command and change the email to something relevant:

gpg --sign-key bob@example.org

The real purpose of this is that if someone else (Alice) sees you’ve ‘signed’ another person’s key (Bob’s), then if Alice trusts you she is more likely to implicitly trust Bob as well. It’s a web of trust.

In order for Bob to benefit from your ‘trust’ you need to send him back the key you’ve signed:

gpg --export --armor bob@example.org

This will send their public key to stdout along with your signature and so they can run the following command to import the signed data you’ve provided:

gpg --import <file path>

Digital Signatures

Earlier, when discussing PKI, we mentioned the use of a MAC which was a form of ‘digital signature’. The benefit of a signature is to allow you to verify that the file (encrypted or plaintext) was indeed created by the person you think it was.

In GPG there are three types of signatures:

  1. clearsigned
  2. detached
  3. attached signatures

For the purposes of this demonstration I should clarify that I have multiple GPG profiles on my laptop, but I’ll refer to two of them as “Bob” and “Alice”. The reason I’m mentioning this is that I’ll use Bob to sign stuff and I’ll use Alice for the verification of the signed data.

In order to verify a signature you need to have the public key for the person who signed the data. Hence utilising multiple GPG profiles makes this easier to demonstrate. So let’s start by creating a secret file:

echo foobar > secret.txt
clearsigned signatures

OK, so let’s begin with Bob creating a clearsign signature:

gpg --local-user Bob --clearsign secret.txt

Once we run this command we can see the file secret.txt.asc was created:

$ cat secret.txt.asc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

foobar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJYIZjsAAoJEMtev7lWHIYbIWkP/jsqtpP7FzWzrS1r4ABgKjG3
QCME6XtISPo8IC2m0BZ57wJcE+zw86P7TigzwenwwNgcVsIMQi/u5TUA5IEp1sUa
IBLk5lMFVWhbJryhgWz+qTGUsnnhu9vZUBjJddNe//Jr2sXlJUPxw+dHnOFGtDpR
90nl+7/gf2+0IdsgX5FstIOfhZLeaeuI4O6NNlwn0GiZW4GirusGpfhP6xrPpYir
GoK5D5v2A0uKfsmWdFA+1wxrk0H5eVghLKN6Ffy2PHIlQaUSzMlexpwzbls0rLlO
UK445AlmZHIWWxc42iwCx9hjdtd+lTg4GXWz3r03iwp9L5pJkaqLFBDpLr3MSdm8
kyMqIevZ9AM3thtQ6T9NFmMVpMGcpjvacy6HxJ0r3eWsQNMXfKcZo1AtcKz0Icqw
tGaO07gcKBdkdwf2+XwTqm/x4LZZFyKsaXJReuQoEWd2+mJJLqgSy4cgfaBKX1e3
YF8ES6RSIeDmxcDaUx9exBkRsj1l0Ot7AYCRmCdYo+CYUBARag3OWswGNwPQ2TLd
c3DoNmbApzPG0V5VTbjarSWuiBCjcY5PfYDGkM2+IOaPL9LmZKDzeEGeaQiJ4k7a
lBqmqvAB6mQ2ZSHGywCUogbg9pwDV6UY93o10QRR8OapmXqW+H0FRkFsH/Sz1yff
J4rLy4J820CAFgUKaUfx
=KdyK
-----END PGP SIGNATURE-----

Alice can now verify the file was signed by Bob (as she has Bob’s public key already):

$ gpg --local-user Alice --verify secret.txt.asc 

gpg: Signature made Tue Nov  8 09:20:44 2016 GMT using RSA key ID 561C861B
gpg: Good signature from "Profile 1 (Bob) <example@integralist.co.uk>" [ultimate]
gpg: WARNING: not a detached signature; file 'secret.txt' was NOT verified!

The downside to using a clearsign signature is three fold:

  1. it’s not encrypted data
  2. it’s not the original file
  3. it’s not compressed

The first is obviously not great, but is OK if you don’t care about the message being visible and you’re only interested in proofing you signed it. The second item is a bigger problem in that you have modified the file to include the signature. Meaning the file is no longer the same, and when signing data you shouldn’t modify the data in order to do the signing. The third item is equally not great depending on the size of the file and having to send a potentially large file over the network.

detached signatures

Let’s look at the second type of signature, a ‘detached signature’:

gpg --local-user Bob --detach-sign secret.txt

So once you run this command you’ll find a secret.txt.sig file has been generated (the secret.txt still hasn’t been encrypted). You can’t see the contents of the .sig file as it has been compressed †

† this is not the same as being ‘encrypted’, it is just compressed for the sake of performance and easier transportation

This is better than --clearsign as the original file hasn’t been modified in order to produce the signature. But this does mean in order for Alice to verify the signature, she needs to have the original plaintext file. If she doesn’t, then you’ll have to send her both the signature and the file.

$ gpg --local-user Alice --verify secret.txt.sig

gpg: assuming signed data in 'secret.txt'
gpg: Signature made Tue Nov  8 09:26:56 2016 GMT using RSA key ID 561C861B
gpg: Good signature from "Profile 1 (Bob) <example@integralist.co.uk>" [ultimate]
attached signatures

Finally, let’s consider ‘attached signatures’. This mechanism enforces the use of verification before you can use the file (hence why the signature is considered attached to the file).

In order for Bob to attach his signature (and thus making verification mandatory) he must use --sign. The attachment aspect is sort of theorectical in that it works by explicitly specifying --encrypt, and so if you didn’t use that flag then this example becomes much like --clearsign but with the small benefit of being compressed.

But if you do use --encrypt with --sign then Alice can’t just use --verify by itself, she has to use --decrypt.

gpg --local-user Bob --encrypt --recipient Alice --sign secret.txt

Again, this generates a secret.txt.gpg file, but this time Alice must enter the passphrase for her private key in order to use the --decrypt flag as the file is no longer just ‘compressed’ it is also ‘encrypted’ as well (--decrypt doesn’t just decrypt the encrypted file but also ‘verifies’ its attached signature):

gpg --local-user Alice --decrypt secret.txt.gpg_private

Which results in the following ouput:

foobar
gpg: Signature made Tue Nov  8 10:13:37 2016 GMT using RSA key ID 561C861B
gpg: Good signature from "Profile 1 (Bob) <example@integralist.co.uk>" [ultimate]

All other uses of --decrypt simply meant decompress the file, hence no passphrase was required. This also explains why Bob had to explicitly specify --recipient when using --encrypt.

Note: in all these examples I use --local-user to change the GPG profile.
But you could also use --default-key if you wanted

Revoking Keys

If you’ve pulled keys from a public server:

gpg --keyserver pgp.mit.edu  --search-keys bob@example.org

Then you want to regularly check those keys are still valid, and haven’t been compromised:

gpg --refresh-keys

Note: you can specify --keyserver when refreshing key data

OpenSSL encryption

The easiest way to encrypt a file using OpenSSL is like so (the password for the file is foobar and is specified at the end of the command; you’ll also see we’re using a -salt with the password to help improve the security):

echo -n 'someTextIWantToEncrypt' | openssl enc -e -salt -out test.txt -aes-256-cbc -pass pass:foobar

To decrypt this file we can use the -d flag (instead of -e which is for encryption):

openssl enc -d -salt -in test.txt -out decrypted.txt -aes-256-cbc -pass pass:foobar

Alternatively you can make things more complex, but ultimately more secure, by encrypting the file using the public key of the person you want to share the file with (which is how GPG works ‘out-of-the-box’). As this means the only person who can decrypt the file is the owner of the private key associated with the public key used for encryption.

In the following example let’s assume you/we are “Alice” and the recipient of the encrypted file is “Bob”:

# Bob needs to send us their RSA key in the PEM certificate format

# So first thing Bob does is generate a PEM containing his private key
openssl rsa -in id_rsa -outform pem > id_rsa.pem

# Second thing Bob does is generate a PEM containing his public key
openssl rsa -in id_rsa -pubout -outform pem > id_rsa.pub.pem

At this point Alice and Bob have to figure out how to securely share Bobs public key with Alice. I would suggest they do this in person to avoid network sniffers getting involved and Alice encrypting the file with the wrong details (this is where PKI helps with “authentication” - we don’t have that process/mechanism here unfortunately).

Once Alice has Bob’s public key she’ll follow these steps:

# Alice generates a 256 bit (32 byte) random key
openssl rand -base64 32 > key.bin

# Alice encrypts the random key using Bobs public key
openssl rsautl -encrypt -inkey id_rsa.pub.pem -pubin -in key.bin -out key.bin.enc

# Alice encrypts the secret file using Bobs public key
openssl enc -aes-256-cbc -salt -in SECRET_FILE -out SECRET_FILE.enc -pass file:./key.bin

Alice can now send the encrypted file (e.g. SECRET_FILE.enc) to Bob. Once Bob has the encrypted file, he’ll follow these steps:

# Bob decrypts the random key using his private key
openssl rsautl -decrypt -inkey id_rsa.pem -in key.bin.enc -out key.bin

# Bob decrypts the file using his private key
openssl enc -d -aes-256-cbc -in SECRET_FILE.enc -out SECRET_FILE -pass file:./key.bin

Keybase

Keybase was mentioned earlier when we were discussing the problem of ‘authentication’ and that there are many public repositories where you can locate a person’s public key. Keybase is a recent attempt at trying to solve this problem in a modern way.

I wont cover all the details of getting setup with Keybase (plus it’s invitation only at the moment), but in essence you need to download their command line tool:

brew install keybase

Once you have that installed you can log into your account:

keybase login

At this point you can use either the website or the command line tool to generate a keypair for your account. If using the command line, then execute the following:

keybase pgp gen

Alternatively you might want to use an already existing private key:

keybase pgp select

Note: the keybase program will push the public key part of your PGP or GPG key pair to the Keybase website and associate it with your Keybase account

The point of Keybase is to help you verify the person you want to communicate with is who they say they are. So Keybase let’s users prove who they are by authenticating with their social accounts. For example, if I search for a friend of mine on Keybase:

keybase search sthulb

Then this will display the following output:

sthulb twitter:sthulb github:sthulb dns://thulbourn.com

Now I know that he is sthulb on GitHub so if the user sthulb was able to verify the GitHub account of the same user to Keybase then I’m pretty sure this is a legit setup and that I’m OK to communicate with this Keybase user.

Now if I wanted to encrypt a file for my friend, then I would do:

keybase encrypt -i secrets.txt -o secrets.txt.asc sthulb

This will then display the following:

▶ INFO Identifying recipient sthulb
✔ public key fingerprint: D38B 2537 67F4 1E59 DFE6 7879 EDD0 4A02 7DD5 BD7A
✔ admin of DNS zone thulbourn.com: found TXT entry keybase-site-verification=JVupJZY1EQbjTiYiLc6JSVNcOWACdBEq9WfuJJWT0qg
✔ "sthulb" on twitter: https://twitter.com/sthulb/status/692285366053834753
✔ "sthulb" on github: https://gist.github.com/13a7654d90750bf154c3

Note: .asc is a convention to indicate a file has been encrypted

My friend would then be able to decrypt the encrypted file I send to him with:

keybase decrypt -i secrets.txt.asc -o secrets.txt

But if someone sends you an encrypted file using your Keybase public key but is encrypted via a different tool, such as their own local GPG or PGP, then you’d decrypt the file with (for example):

keybase pgp decrypt -i some_encrypted_file.gpg

If you want to encrypt a file for someone who doesn’t use Keybase (e.g. they use their own local GPG installation), then you can export your public/private key from Keybase using the command line tool and then import them into your local GPG so you can utilise GPG to encrypt your data and specify the user’s public key:

keybase pgp export -o keybase.public.key
keybase pgp export -s -o keybase.private.key
gpg --import keybase.public.key
gpg --allow-secret-key-import --import keybase.private.key

Notice the use of -s to export the private key

Now you can encrypt data via GPG using your Keybase private key:

echo foobar > secret.txt
gpg --encrypt -u 123 -r 456 secret.txt

Note: 123 being your keybase identifier inside GPG and
456 being the recipient identifier
(assuming you’ve imported their public key already)

The recipient can now decrypt the file:

gpg -d secret.txt.gpg

Which should I use?

OpenSSL is not considered secure enough in today’s digital age. There are known bugs with the implementation of the OpenSSL enc command, and so the recommendation in the security community seems to be to ditch OpenSSL for GPG (at least for these types of scenarios where we’re simply encrypting a file that we want to share with someone else).

Creating, self-signing, issuing and revoking certificates

OK, so I was going to go through the process of creating a new CA root and then self-signing the certificate so we can then go ahead and issue certificates from our own personal CA. The idea was to indicate how you might do this for an organisation that doesn’t want to pay for a CA to provide them a certificate (e.g. services that only allow access via client certificates doesn’t have to worry about being trusted; as long the employees have trusted the organisation’s self-signed certificate then it’s fine).

The reason I’m not going to do that is because Ivan Ristić (author of “Bulletproof SSL and TLS”) has already done the leg work and has made it freely available in his ebook “OpenSSL Cookbook”. Which you can get here https://www.feistyduck.com/books/openssl-cookbook/

Although, if you want a super quick run down…

You can generate a CSR (Certificate Signing Request; which you send to a CA to approve) using

openssl req -sha256 -new -key my-private-key.pem -out csr.pem

You can then self-sign that certificate (while you wait for the CA to officially create you a cert and sign it) using:

openssl x509 -req -days 365 -in csr.pem -signkey my-private-key.pem -out my-certificate.pem

Note: create key pair + cert in a one liner
openssl req -nodes -new -x509 -keyout server.key -out server.cert

UPDATE

OK so I ended up writing about this any way in this article where I discuss how to handle Client Certificate Authentication using Docker.

Conclusion

Hopefully, you’ve stuck with me until the end here and that you found the information contained useful, or dare say even enlightening. I wrote this post to help solidify my own knowledge and for this to become a future reference point for myself; but I ended up really enjoying diving into aspects such as the PKI and SSL handshake process, as it’s an area that has confused me for the longest time.

If there are any glaring mistakes (I’m sure there will be a few) then do please let me know so I can update and correct.

References