CertPortal: Building Self-Service Secure S/MIME Provisioning Portal

tl;dr

NCC Group’s Research Development team designed and built CertPortal which allows users to create and manage S/MIME certificates automating the registration and renewal to allow enterprise scale deployment.

The core of the system integrates DigiCert to create an S/MIME certificate and then storing both the certificate, the password, creation and expiry dates in a CyberArk safe. It then publishes the public certificate in the Microsoft Exchange Global Address List against the user’s account.

The portal presents the user with the two options of ‘show me my password’ and ‘download certificate’. This approach has removed a number of manual processes whilst providing significant efficiency and security gains.

The Beginning

Encryption as standard is both a crowning jewel and a backbone of modern HTTP traffic, so much so that browsers warn users of websites that do not offer it. On top of that, services like Let’s Encrypt make the process of adding encryption to a public facing website easy.

In the world of S/MIME this is not the case. There are services such as DigiCert or Entrust which allow API users to automatically generate S/MIME certificates but this still leaves a large earnest on the IT team to run the scripts to generate a password and private key, request a certificate, then securely deliver all of the above to the end user.

In steps CertPortal, an S/MIME self-service portal to fill this void. It needs to be able to do three things well:

1. On request generate a password, private key, and CSR, request a certificate, and generate a PFX file.
2. Securely store those files and retrieve them on request.
3. Provide a simple-to-use interface for users to perform the first two things.

In this post we will be discussing the first thing: Generating an S/MIME certificate.

Passwords, Private Keys, and CSRs

Assuming we have received a request to generate an S/MIME certificate the first thing we need to do is prepare ourselves to request a certificate from our Certificate Authority. This requires us to create a cryptographically secure password, a private key, and a Certificate Signing Request (CSR).

Secure Passwords

The first step is to generate a secure password, fortunately for us since Python 3.6 the secrets module has been available which makes this process fairly straight forward. We want our script to be repeatable so we will store our generated password on local disk so if we have a failure later on in the pipeline we can easily restart the process to try again.

This means we need to check our storage directory for an existing password file and return the contents if found. Otherwise we need to generate a cryptographically secure password, store it in the storage directory, and return the newly generated password.

import os
import secrets

PWD_CHRS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^ *-+=,."
PWD_LEN = 16


def get_password(storage_dir: str) -> bytes:
    pwd_file = os.path.join(storage_dir, 'password.txt')
    try:
        with open(pwd_file, 'rb') as f:
            password = f.read()
    except FileNotFoundError:
        list_rand_chrs = [secrets.choice(PWD_CHRS) for _ in range(PWD_LEN)]
        password_str = ''.join(list_rand_chrs)
        password = password_str.encode('utf-8')
        with open(pwd_file, 'wb') as f:
            f.write(password)

    return password

The interesting line from the above excerpt is generating the password. Choosing the password characters and length is a matter for your own security policies. There are three steps to generating the password:

1. Generate a list of random characters: secrets.choice(PWD_CHRS) for _ in range(PWD_LEN)
2. Convert the list into a string ''.join(list_rand_chrs).encode('utf-8')
3. Convert the list into its bytes representation: ''.join(list_rand_chrs).encode('utf-8')

Private Keys

Once we have a password the next step is to generate a private key. To achieve this we will use the cryptography package. Like the previous step we want this step to be robust and repeatable.

This means we need to check our storage directory for an existing private key file, loading it using the password, and return it. Otherwise we generate a new private key with the password, store it in the storage directory, and return the private key.

import os

from cryptography.hazmat.primitives.serialization import load_pem_private_key
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization

KEY_SIZE = 4096


def get_private_key(storage_dir: str, password: bytes):
    key_file = os.path.join(storage_dir, 'private.key')
    try:
        with open(key_file, 'rb') as f:
            key = load_pem_private_key(
                f.read(), password, default_backend())
    except (ValueError, FileNotFoundError):
        key = rsa.generate_private_key(
            public_exponent=65537,
            backend=default_backend(),
            key_size=KEY_SIZE)

        with open(key_file, 'wb') as f:
            f.write(key.private_bytes(
                encoding=serialization.Encoding.PEM,
                format=serialization.PrivateFormat.TraditionalOpenSSL,
                encryption_algorithm=serialization.BestAvailableEncryption(password)))

    return key

This process is fairly straight forward when you know what you’re doing. The important part is choosing the key size which needs to be a power of 2 (at least 2048).

Certificate Signing Requests

The last part of this step is creating the Certificate Signing Request (CSR). To achieve this simply follow the tutorial provided by the cryptography package with a couple of minor amendments:

csr = x509.CertificateSigningRequestBuilder().subject_name(x509.Name([
    # Provide various details about who we are
    x509.NameAttribute(NameOID.COMMON_NAME, csr_attrs['common_name']),
    x509.NameAttribute(NameOID.COUNTRY_NAME, csr_attrs['country_name']),
    x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, csr_attrs['state_name']),
    x509.NameAttribute(NameOID.LOCALITY_NAME, csr_attrs['locality_name']),
    x509.NameAttribute(NameOID.ORGANIZATION_NAME, csr_attrs['organization_name']),
    x509.NameAttribute(NameOID.ORGANIZATIONAL_UNIT_NAME, csr_attrs['organization_unit_name']),
])
).add_extension(x509.SubjectAlternativeName([
    x509.DNSName(csr_attrs['common_name'])
]), critical=False
).sign(key, hashes.SHA512(), cryptography_default_backend())

We need to set the ‘Common Name’ and subject ‘Alternative Name’ as the email address the S/MIME is being generated for. There is no need to check the storage directory for an existing CSR as, unlike the password and private key, the CSR will be the same every time (if we use the same private key).

Asking the Certificate Authority

Now that we have a password, private key, and CSR we can move onto asking the Certificate Authority (CA) for an S/MIME certificate. This step will vary depending on which provider we are going to use but the basic process is the same for all:

1. Authenticate with their API
2. Request a new S/MIME type certificate from their API
3. Store the request ID returned for future reference
4. Waiting until the S/MIME certificate has been issued
5. Download the certificate (including all chain certificates)
6. Store the certificates in the same storage directory as the password, private key, and CSR

Bundle into a PFX (PKCS12) file

Lastly, we need to bundle the private key, S/MIME certificate, and chain certificates into a PFX (PKCS12) file for our end user to be able to download via the portal. To do this we will require one more Python package: OpenSSL (note that the cryptography package has since added PKCS12 serialization since version 3, however CertPortal currently uses version 2). Like the CSR, the process here is straightforward when you know what you are doing:

from OpenSSL import crypto


def get_pfx(storage_dir: str, cert, chain: list, key, password: bytes, friendly_name: bytes = None):
    """
    Generate a PFX (PKCS12) from the certificate(s), key, and password then store it in a file
    called `smime.pfx` inside `storage_dir`.
    :param storage_dir: Directory to store the PFX file
    :param cert: cryptography S/MIME certificate
    :param chain: List of cryptography chain certificates issued by the CA
    :param key: The cryptography private key
    :param password: The private key password
    :param friendly_name: The friendly name in the PKCS #12 structure
    """
    pfx = crypto.PKCS12()
    pfx.set_friendlyname(friendly_name or b'S/MIME Certificate')
    pfx.set_privatekey(crypto.PKey.from_cryptography_key(key))
    pfx.set_certificate(crypto.X509.from_cryptography(cert))
    pfx.set_ca_certificates([crypto.X509.from_cryptography(c) for c in chain])
    pfx_data = pfx.export(password)

    with open(os.path.join(storage_dir, certificate_file_name('pfx')), 'wb') as f:
        f.write(pfx_data)

    return pfx

We start by creating a crypto.PKCS12 object and setting a friendly name for the certificate. Next we set the private key. Then we set the S/MIME certificate (after converting from the cryptography key) and the same for the CA certificates. Finally we export the data and store it in the storage directory.

Conclusion

At the end of this process we have a secure password, private key, CSR, and PFX ready to be downloaded by the end user. We still need to be able to securely store these files and provide an interface for an end user to generate a new certificate and access existing certificates. We will address the secure storage in a future post.

We hope that this post has given you a good understanding of the how to create an S/MIME certificate progamtically.

Final remarks

We have only looked at the bare bones of this process and as such skipped over many best practices such as storing the configurations, using interfaces and implementing CA specific backends, etc. These are, however, outside the scope of the post.