Table of Contents for

Network Security with OpenSSL

The low-level interface to Diffie-Hellman provided by OpenSSL consists of a structure of the type DH and a set of functions that operate on that structure. The DH structure and functions are made accessible by including the openssl/dh.h header file. The DH structure itself contains many data members that are of little or no interest to us, but four members are important, as shown in the following abbreviated DH structure definition:

typedef struct dh_st
{
    BIGNUM *p;
    BIGNUM *g;
    BIGNUM *pub_key;
    BIGNUM *priv_key;
} DH;

The p and g members, known as Diffie-Hellman parameters, are public values that must be shared between the two parties using the algorithm to create a shared secret. Because they’re public values, no harm will come from a potential attacker discovering them, which means that they can be agreed upon beforehand or exchanged over an insecure medium. Typically, one side of the conversation generates the parameters and shares them with the peer.

The p member is a prime number that is randomly generated. For temporary keys, it is typically at least 512 bits in length, whereas persistent keys should more appropriately be at least 1,024 bits. These sizes correspond to the notion of ephemeral and static keys presented in Chapter 5. The prime that will be used for p is generated such that (p-1)/2 is also prime. Such a prime is known as a safe or strong prime . The g member, also known as the generator , is usually a small number greater than one. OpenSSL functions best when this number is either two or five. A value of two is sometimes used for performance reasons, but keep in mind that faster key generation also means that an attacker can break the algorithm faster. In general, we recommend that a value of five be used.

Using the two public parameters, p and g, each pair chooses a random large integer for the priv_key member. A value for the pub_key member is computed from the priv_key member and shared with the peer. It is important that only the value of the pub_key member be shared. The value of the priv_key member should never be shared.

Using the values of priv_key and the peer’s pub_key, each peer can independently compute the shared secret. The shared secret is suitable for use as the key for a symmetric cipher. The entire exchange between the peers can be done over an insecure medium. Even if someone captures the parameter and key exchange, the attacker will not be able to determine the shared secret.

Diffie-Hellman requires that both parties involved in the key exchange use the same parameters to generate public keys. This means the parameters either need to be agreed upon and exchanged before the conversation begins, or the parameters must be generated and exchanged as part of the key exchange. Either way, the parameters must first be generated by one party and given to the other, or perhaps generated by a third party and given to both. For the purposes of this discussion, we will assume that the generation of the parameters will be done as part of the key exchange process, although this is often not desirable because it can take a significant amount of time to generate the parameters.

The participants in the key agreement must first agree which party will be responsible for generating the parameters that they’ll both use. In a client/server scenario, the parameters are usually generated by the server. Often, the server will generate the parameters when it starts up or retrieve them from a file that contains already generated parameters, and use the same parameters for each client that connects. It is also common for both the client and server to have a copy of the parameters built into the respective applications.

OpenSSL provides the function DH_generate_parameters , which will create a new DH object that is initialized with fresh values for p and g. The generation of the parameters generates a value only for p. The value of g is specified by the caller and is not chosen randomly by OpenSSL. The value of g should be a small number greater than one, usually either two or five.

DH *DH_generate_parameters(int prime_len, int generator,
                            void (*callback)(int, int, void *), void *cb_arg);

Using the generation function alone can be dangerous. While the generation function does have validity checks for the prime that it generates, it could generate a prime that is not suitable for use with the algorithm. For this reason, the function DH_check should always be used to ensure that the generated prime is suitable.

int DH_check(DH *dh, int *codes);

If the function encounters an error unrelated to the validity of the generated prime, the return will be zero; otherwise, it will be nonzero. When the function returns successfully, the codes argument will contain a bit mask that indicates whether the parameters are suitable for use or not. If none of the bits is set, the parameters should be considered suitable for use. If any of the following bits are set, the parameters may not be suitable for use. In most cases, the parameters should be thrown away, and new ones should be generated.

Once the parameters have been generated, they can be transmitted to the peer. The details of how the data is sent depend on the medium that is being used for the exchange. To transmit the parameters over a TCP connection, the BIGNUM functions BN_bn2bin and BN_bin2bn are obvious candidates.

The party that generates the parameters calls DH_generate_parameters to obtain a DH object. The party that is receiving the parameters must also obtain a DH object. This is easily done by calling the function DH_new , which will allocate and initialize a new DH object. The parameters that are received from the peer can then be directly assigned to the DH object’s p and g data members, using the appropriate BIGNUM functions.

When we’re done with a DH object, we must be sure to destroy it by calling the function DH_free , and passing the pointer returned by either DH_generate_parameters or DH_new as the only argument.

Now that parameters have been generated and received by the two peers, each peer must generate a key pair and exchange their public keys. Remember that the private key must not be shared at all. Once this is done, each peer can independently compute the shared secret, and the algorithm will have done its job. With authenticated Diffie-Hellman, the public/private key pairs can persist beyond usage for a single key-agreement. In these cases, we must be wary of a special class of attack against Diffie-Hellman, which is discussed at the end of this section.

OpenSSL provides the function DH_generate_key for generating public and private keys. It requires as its only argument a DH object that has the parameters, p and g, filled in. If the keys are generated successfully, the return from the function will be nonzero. If an error occurs, the return will be zero.

Once the keys have been generated successfully, each peer must exchange their public key with the other peer. The details of how to exchange the value of the public key varies depending on the medium that is being used, but in a typical case in which the communication is taking place over an established TCP connection, the functions BN_bn2bin and BN_bin2bn will once again work for the exchange of the DH object’s pub_key data member.

With the parameters and public key now exchanged, each party in the exchange can use his own private key and the peer’s public key to compute the shared secret using the function DH_compute_key .

int DH_compute_key(unsigned char *secret, BIGNUM *pub_key, DH *dh);

After the shared secret is computed, the DH object is no longer needed unless more secrets will be generated and exchanged. It can be safely destroyed using the DH_free function.

In certain cases, Diffie-Hellman can be subject to a type of attack known as a small-subgroup attack . This attack results in a reduction of the computational complexity of brute-forcing the peer’s private key value. Essentially, a small-subgroup attack can result in the victim’s private key being discovered. There are several different methods of protecting Diffie-Hellman against this type of attack. The simplest method is to use ephemeral keying. If both parties stick to ephemeral keying and use a separate method of authentication, small-subgroup attacks are thwarted. This isn’t always feasible, however, mostly due to computational expense. If static keys will be used, two simple mathematical checks can be performed on the public key received from a peer to ensure these attacks aren’t possible. If the key passes both tests, it’s safe to use. The first test verifies that the supplied key is greater than 1 and less than the value of the p parameter. The second test computes y ^q mod p, in which y is the key to test and q is another large prime. If the result of this operation is 1, the key is safe; otherwise, it is not. The q parameter is not generated by OpenSSL even though there is a placeholder for it in the DH structure. An algorithm for generating q can be found in RFC 2631. If you’re interested in the other methods or more detailed information on the attack, we recommend that you read RFC 2785.

When we began our discussion of Diffie-Hellman, we mentioned that it provides key agreement and authentication. Use of the authentication features of this protocol is not very common; thus, pairing Diffie-Hellman with another algorithm for authentication is often done. The threat is that mistakenly leaving out authentication can lead to susceptibility to man-in-the-middle attacks. To execute such an attack, the attacker sits in between two hosts that are trying to communicate and intercepts all of the messages. For example, suppose that Alice and Bob plan to use Diffie-Hellman to make a shared secret. Charlie could intercept all messages from Alice to Bob and all messages from Bob to Alice. From this position, Charlie can agree upon a key with Alice and a different key with Bob. When the attacker receives a message from Alice, he decrypts it with the key he negotiated with her and reads the message. He can then encrypt the message using the key he negotiated with Bob and pass it along to him. Alice and Bob will believe that they’re communicating securely. They’ll be completely unaware that Charlie is eavesdropping and worse, possibly even altering their messages, inserting forged messages, or not passing the messages along at all.

To alleviate this problem, Diffie-Hellman should always be used with some method of authentication, most commonly from another algorithm. This is accomplished by authenticating the messages containing public values for the Diffie-Hellman agreement. Using signatures, each party would exchange their public keys to use for signing before the conversation begins, and then sign the public value before sending it. The details will be explained in the following section.

Similar to the low-level interface to Diffie-Hellman, the low-level interface to DSA provided by OpenSSL consists of a DSA structure and a set of functions that operate on that structure. The DSA structure and functions are made accessible by including the openssl/dsa.h header file. The DSA structure itself contains many data members that are of little or no interest to us, but five members are important, as shown in the following abbreviated DSA structure definition:

typedef struct dsa_st
{
    BIGNUM *p;
    BIGNUM *q;
    BIGNUM *g;
    BIGNUM *pub_key;
    BIGNUM *priv_key;
} DSA;

The p, q, and g members, known as DSA parameters, are public values that must be generated before a key pair can be generated. Because they’re public values, no harm will come if a potential attacker discovers them. The same parameters can be safely used to generate multiple keys. In fact, RFC 2459 specifies a mechanism in which DSA parameters for a certificate can be inherited from the certificate of the issuer. Using parameter inheritance not only reduces the size of certificates, it also enforces the sharing of parameters.

The p member is a prime number that is randomly generated. Initially, the proposed standard fixed the length of the prime at 512 bits. Due to much criticism, this was later changed to allow a range between 512 and 1,024 bits. The length of the prime must be a multiple of 64 bits, however. OpenSSL does not enforce the 1,024-bit upper bound, but it’s not a good idea to use a prime larger than 1,024 bits—many programs may not be able to use the keys that result from such a large prime. The q member is a prime factor of p-1. The value of q must also be exactly 160 bits in length. The g member is the result of a mathematical expression involving a randomly chosen integer, as well as p and q.

Using the three public parameters (p, q, and g), the public and private keys can be computed. Public parameters used to compute the keys are required to generate or verify a digital signature. The parameters must therefore be exchanged along with the public key in order for the public key to be useful. Of course, the private key should never be distributed.

Like Diffie-Hellman, DSA requires generation of parameters before keys can be generated. Once a set of parameters is generated, many keys can be created from it. This does not mean that only people with keys created from the same parameters can interoperate; it simply means that in order to do so, all of the parameter values must be included as part of the public key.

The interface for generating DSA parameters is similar to the interface for generating Diffie-Hellman parameters. The function DSA_generate_parameters will create a new DSA object that is initialized with fresh values for p, q, and g. Generation of DSA parameters is more complex than Diffie-Hellman parameter generation, although it is typically much faster, especially for large key lengths.

DSA * DSA_generate_parameters(int bits, unsigned char *seed, int seed_len,
                              int *counter_ret, unsigned long *h_ret,
                                void (*callback)(int, int, void *), void *cb_arg);

Once parameters have been generated, key pairs can be generated. The function DSA_generate_key is provided by OpenSSL for doing just that. It requires as its only argument a DSA object that has the parameters p, q, and g filled in. If the keys are generated successfully, the return from the function will be nonzero. If an error occurs, the return will be zero. The generated public and private keys are stored in the supplied DSA object’s pub_key and priv_key members, respectively.

Once the keys are computed, they can be used for creating digital signatures and verifying them. The DSA object containing the parameters and keys must remain in memory to perform these operations, of course, but when the DSA object is no longer needed, it should be destroyed by calling DSA_free and passing the DSA object as its only argument.

Digital signatures must be computed over a message digest of the data that will be signed. The DSA standard dictates that the message digest algorithm used for this is SHA1, which has output of the same size as the DSA q parameter. This is no coincidence. In fact, the DSA algorithm cannot be used to compute a signature for data that is larger than its q parameter.

OpenSSL provides a low-level interface for computing signatures that does allow you to use DSA to sign arbitrary data. By this, we mean that OpenSSL does not enforce the requirement to use SHA1 as the message digest to sign, or even to sign a message digest at all! We strongly recommend against using the low-level interface for signing and verifying in favor of using the EVP interface, which we describe later in this chapter. However, we will briefly cover the three low-level functions for creating and verifying DSA signatures.

Computing a digital signature can be a processor-intensive operation. For this reason, OpenSSL provides a function that allows for the pre-computation of the portions of a signature that do not actually require the data to be signed.

int DSA_sign_setup(DSA *dsa, BN_CTX *ctx, BIGNUM **kinvp, BIGNUM **rp);

Care must be taken when pre-computing the kinv and r values for signing. Both values must be placed in the DSA object. If only one is placed in the object, the memory used by the other will be leaked. Additionally, the values cannot be reused. They will be destroyed by the signing function. It may seem tempting to save a copy of the pre-computed values and reuse them for signing multiple pieces of data, but you must not. An integral part of a DSA signature is a 160-bit (actually, the same size as the q parameter, but that should always be 160 bits) random number known as k. If the same value for k is used more than once, it is possible for an attacker to discover the private key.

The function DSA_sign is provided for actually computing a digital signature using the DSA algorithm. If pre-computation of kinv and r has not been done, the function will perform the computation itself.

int DSA_sign(int type, const unsigned char *dgst, int len,
             unsigned char *sigret, unsigned int *siglen, DSA *dsa);

The function DSA_verify is used to verify signatures and is similar to the function used to create them.

int DSA_verify(int type, const unsigned char *dgst, int len,
               unsigned char *sigbuf, int siglen, DSA *dsa);

DSA does not provide a mechanism for key agreement. It cannot create a shared secret for us, whereas an algorithm such as Diffie-Hellman can. Thus, it is common to employ a protocol that uses Diffie-Hellman and DSA to deliver two-way authentication.

First, we need to establish that the client and server each have a public/private key pair and can exchange public keys offline, prior to beginning secure communications. At the same time, we need to assume they agree upon parameters for Diffie-Hellman key agreement. Overall, these assumptions are not egregious. A protocol might begin with each party computing public and private Diffie-Hellman keys. At this point, instead of sending that unauthenticated data to the opposite end, each party will sign the message containing the public value with their own private DSA key. After this is done, they can send the signed messages.

The value of that signature is that it assures both parties that the public keys actually came from their respective peer because only the server has the server’s private key, and only the client has the client’s private key. If both parties can verify the signature on the public key they’ve received, they can be sure that there is no man-in-the-middle. What we’ve just described is known as two-way authentication, because each party independently verifies the other.

Just like the low-level interfaces to Diffie-Hellman and DSA, the low-level interface to RSA provided by OpenSSL consists of an RSA structure and a set of functions that operate on that structure. The RSA structure and functions are made accessible by including the openssl/rsa.h header file. The RSA structure itself contains many data members that are of little or no interest to us, but five members are important, as shown in the following abbreviated RSA structure definition:

typedef struct rsa_st
{
    BIGNUM *p;
    BIGNUM *q;
    BIGNUM *n;
    BIGNUM *e;
    BIGNUM *d;
} RSA;

The p and q members are both randomly chosen large prime numbers. These two numbers are multiplied together to obtain n, which is known as the public modulus. References to the strength of RSA actually refer to the bit length of the public modulus. Once the private key has been computed, p and q may be discarded, but they should never be disclosed. However, keeping the values for p and q around is a good idea; having them available increases the efficiency with which private key operations are performed.

The e member, also known as the public exponent, should be a randomly chosen integer such that it and (p-1)(q-1) are relatively prime . Two numbers are relatively prime if they share no factors other than one; they may or may not actually be prime. The public exponent is usually a small number, and in practice, is usually either 3 or 65,537 (also referred to as Fermat’s F4 number). Using e, p, and q, the value of d is computed.

Together, the n and e members are the public key, and the d member is the private key.

The RSA algorithm does not require parameters to generate keys, which makes key generation a much simpler affair. OpenSSL provides a single function to create a new RSA key pair, which will create a new RSA object that is initialized with a fresh key pair.

RSA *RSA_generate_key(int num, unsigned long e,                
                      void (*callback)(int, int, void *), void *cb_arg);

Once keys are generated, it is a good idea to call RSA_check_key to verify that the keys generated by RSA_generate_key are actually usable. The function requires an RSA object as its only argument. The RSA object should be completely filled in, including values for its p, q, n, e, and d members. If the return value from the function is 0, it indicates that there is a problem with the keys, and that they should be regenerated. If the return value from the function is 1, it indicates that all tests passed, and that the keys are suitable for use. If the return value from the function is -1, an error occurred in performing the tests.

When the RSA key is no longer needed, it should be freed by calling RSA_free and passing the RSA object as its only argument.

Now that we’ve explored key setup, we can look at the different methods of using those keys. The RSA algorithm allows for secrecy because it can encrypt data. Data encrypted with a public key can be decrypted only by an entity possessing the corresponding private key.

Before looking at the specifics of these operations, we must first discuss blinding . Any RSA operation involving a private key is susceptible to timing attacks. Given an RSA operation as a black box that we feed data into and collect results from, an attacker can discern information about our key material by measuring the amount of time it takes the various operations to complete. Blinding is essentially a change in the implementation that removes any correlation between the amount of time taken for an operation and the private key value.

The function RSA_blinding_on enables blinding for the RSA object passed into it as the first argument. This means that any operation on the RSA object involving the private key will be guarded from timing attacks. The second argument is an optional BN_CTX object, which may be specified as NULL. Likewise, the function RSA_blinding_off disables blinding for the RSA object passed into it. If you ever design a system that allows arbitrary operations on a private key, such as any system that automatically signs data, enabling blinding is important for the safety of the private key.

With RSA, it is most common to perform encryption with a public key and decryption with the corresponding private key. The functions RSA_public_encrypt and RSA_private_decrypt provide the means to perform these operations. It is also possible to perform encryption with a private key and decryption with the corresponding public key. Two functions, RSA_private_encrypt and RSA_public_decrypt , are provided by OpenSSL. They’re intended for those who wish to implement signing at a very low level. In general, they should be avoided. Each of the four functions returns the number of bytes, including padding, that were encrypted or decrypted, or -1 if an error occurs.

int RSA_public_encrypt(int flen, unsigned char *from, unsigned char *to,
                       RSA *rsa, int padding);

int RSA_private_decrypt(int flen, unsigned char *from, unsigned char *to,
                         RSA *rsa, int padding);

The encryption performed by RSA requires that the data to be encrypted be appropriately formed. If the data to be encrypted does not fit the requirements, it must be padded. OpenSSL supports several types of padding for RSA encryption:

As with DSA, RSA computes signatures over a chunk of data by first cryptographically hashing the data to obtain a digest value. This digest and the signer’s private key are then used as input to the signing process. Verification can be performed by executing the same hash and using this digest value along with the signature value and the signing party’s public key.

int RSA_sign(int type, unsigned char *m, unsigned int m_len,   
             unsigned char *sigret, unsigned int *siglen, RSA *rsa)

You will notice that the signature of the function is identical to the DSA signing function; however, the arguments are interpreted differently. The same is true for the signature verification function.

int RSA_verify(int type, unsigned char *m, unsigned int m_len,
               unsigned char *sigbuf, unsigned int siglen, RSA *rsa);

It may be tempting to consider implementing a system in which we rely solely on RSA for secrecy. This is actually not a good idea for a variety of reasons, but most importantly because RSA is very slow compared to symmetric ciphers. Additionally, because we can encrypt only small chunks of data at a time with RSA keys alone, the best use for RSA’s data encryption capabilities is for key exchange. There are several ways that RSA can be used in this manner.

While we’ve scratched the surface of the theory involved in designing secure protocols, we advise against trying to design your own, because doing it correctly and securely is very hard. Using a well-known and verifiably secure protocol such as SSL or TLS is highly recommended for all but academic situations.

The EVP interface provides a means to create and verify digital signatures using either DSA or RSA keys. Creating a digital signature requires a private key, while verifying one requires the public key that corresponds to the private key used to create the signature. When a digital signature is created for a piece of data, that same data is required to verify the signature.

The first step in creating a digital signature using the EVP interface is to initialize an EVP_MD_CTX object, discussed in Chapter 6. An EVP_MD_CTX object can be allocated dynamically using malloc or new, or it can be allocated statically as either a global or a local variable. In either case, EVP_SignInit must be called to initialize the object.

int EVP_SignInit(EVP_MD_CTX *ctx, const EVP_MD *type);

The “engine” release and Version 0.9.7 of OpenSSL deprecate the EVP_SignInit function. Instead, EVP_SignInit_ex should be used, which adds a third argument. The additional argument is a pointer to an engine object that has been initialized. It may also be specified as NULL, which causes the default software implementation to be used.

Once a context has been initialized, the data to be signed must be fed into it. This is done by calling EVP_SignUpdate as many times as necessary to feed in all the data. The function works identically to EVP_DigestUpdate and, in fact, is actually implemented as a preprocessor macro that simply calls EVP_DigestUpdate.

int EVP_SignUpdate(EVP_MD_CTX *ctx, const void *buf, unsigned int len);

Once all of the data to be signed has been added into the context, the signature can be computed. This is done by calling EVP_SignFinal . Once EVP_SignFinal has been called, the context is no longer valid, and must be reinitialized with EVP_SignInit or EVP_SignInit_ex before it can be used to create another signature. Note that this does not mean that a dynamically allocated context object has been freed. OpenSSL has no way of knowing whether the context object was dynamically allocated or not, so it is up to the caller to free that memory as appropriate.

int EVP_SignFinal(EVP_MD_CTX *ctx, unsigned char *sig, unsigned int *siglen,
                  EVP_PKEY *pkey);

EVP_SignFinal will return zero if an error occurred in computing the signature, and nonzero if the signature computed successfully.

Verifying a signature is as simple as computing one. The set of functions to verify a signature match the functions to compute a signature, save for their names. Before a signature can be verified, a context must be initialized by calling EVP_VerifyInit . The “engine” release and Version 0.9.7 of OpenSSL deprecate EVP_VerifyInit in favor of EVP_VerifyInit_ex , which requires a third argument that is an engine object or NULL to use the default software implementation of the chosen message digest algorithm.

int EVP_VerifyInit(EVP_MD_CTX *ctx, const EVP_MD *type);

With an initialized context, EVP_VerifyUpdate should be called as many times as necessary to supply the context with all of the data that was purportedly used to create the signature. The data for verifying a signature should be the same data used to create the signature. If the signature and the data do not match, the verification will fail. This, of course, is the whole point of a digital signature—to verify that the data is intact and has not been modified in any way.

int EVP_VerifyUpdate(EVP_MD_CTX *ctx, const void *buf, unsigned int len);

Once all of the data has been passed into the context along with EVP_VerifyUpdate, EVP_VerifyFinal must be called to perform the actual verification of the signature. Its signature matches that of EVP_SignFinal, but its arguments and return value are interpreted differently.

int EVP_VerifyFinal(EVP_MD_CTX *ctx, unsigned char *sigbuf,
                    unsigned int siglen, EVP_PKEY *pkey);

EVP_VerifyFinal will return -1 if an error occurs in verifying the signature. If the signature does not match the data, the return value will be 0, and if the signature is valid, the return value will be 1.

The EVP interface also provides an interface for enveloping data using RSA keys. Data enveloping is the process of encrypting a chunk of data with RSA, typically for securely sending it to a recipient. Initially, it may seem that enveloping is equivalent to using RSA to encrypt all of our data, but this is not correct. Because public key algorithms are inappropriate for encrypting large amounts of data, enveloping requires the sender to generate a random key, also called a session key, and encrypt that key using the public key of the intended recipient. The actual data is then encrypted with a symmetric cipher using the session key. Incorrectly implementing this process often causes bugs or vulnerabilities in programs. To avoid this, OpenSSL provides features in the EVP interface referred to as the “envelope” encryption/decryption interface that handles all of the subtleties correctly.

One of the features offered by the EVP interface for data encryption is the ability to encrypt the same data using several public keys. A single session key is generated and encrypted using each public key that is supplied. The recipients can then use their respective private keys to decrypt the session key, and thus decrypt the data using the appropriate symmetric cipher. The support for this feature is wholly contained by the context initialization function, which means the interface for encrypting for multiple recipients is the same as encrypting for a single recipient.

The act of encrypting data using a public key through the EVP interface is called sealing. A set of functions, EVP_SealInit, EVP_SealUpdate, and EVP_SealFinal, are provided to encrypt an arbitrary amount of data. Each of these functions requires an EVP_CIPHER_CTX object to maintain state. This object can be allocated either dynamically or statically and must be initialized prior to use. Initialization of the context object is achieved by calling EVP_SealInit. Unlike the other EVP context initialization functions that we’ve discussed, EVP_SealInit has not been deprecated in the “engine” release or Version 0.9.7 of OpenSSL.

int EVP_SealInit(EVP_CIPHER_CTX *ctx, EVP_CIPHER *type,        
                 unsigned char **ek, int *ekl, unsigned char *iv,
                 EVP_PKEY **pubk, int npubk);

Example 8-1 demonstrates how to call EVP_SealInit .

ek = (unsigned char **)malloc(sizeof(unsigned char *) * npubk);
ekl = (int *)malloc(sizeof(int) * npubk);
pubk = (EVP_PKEY **)malloc(sizeof(EVP_PKEY *) * npubk);
 
for (i = 0;  i < npubk;  i++)
{
    pubk[i] = EVP_PKEY_new(  );
    EVP_PKEY_set1_RSA(pubk[i], rsakey[i]);
    ek[i] = (unsigned char *)malloc(EVP_PKEY_size(pubk[i]));
}
 
EVP_SealInit(ctx, type, ek, ekl, iv, pubk, npubk);

Once initialization has been performed, the remainder of the sealing process is very much like encrypting with a symmetric cipher, as described in Chapter 6. In fact, it is identical, with the exception of the names of the functions that are used. The initialization function takes care of generating the session key and encrypting it using the public key of each recipient. It also prepares the context to perform the encryption of the actual data using the selected symmetric cipher.

int EVP_SealUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl,
                   unsigned char *in, int inl);

After the data to be encrypted has been fed into EVP_SealUpdate, EVP_SealFinish must be called to finish the job. It will perform any necessary padding and write any remaining encrypted data into its output buffer. Once EVP_SealFinish has been called, EVP_SealInit must be called again before the context can be reused.

int EVP_SealFinal(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl);

When the seal process is complete, the encrypted session key, initialization vector (if any), and encrypted data must all be sent to the recipient in order for the recipient to decrypt the data successfully. The recipient can then use the EVP interface to unseal or open (decrypt) the data.

The function EVP_OpenInit initializes a context for decryption. Like EVP_SealInit, it has not been deprecated by the “engine” release or Version 0.9.7 of OpenSSL.

int EVP_OpenInit(EVP_CIPHER_CTX *ctx, EVP_CIPHER *type, unsigned char *ek,
                 int ekl, unsigned char *iv, EVP_PKEY *pkey);

Initializing the context decrypts the session key and prepares the context for decrypting the data using the specified symmetric cipher. The rest of the process is identical to decrypting data that has been encrypted with a symmetric cipher, i.e., the functions EVP_OpenUpdate and EVP_OpenFinal are identical to the functions EVP_DecryptUpdate and EVP_DecryptFinal in every way, except for their names.

int EVP_OpenUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl,
                   unsigned char *in, int inl);

Once all of the data to be decrypted has been fed into EVP_OpenUpdate, EVP_OpenFinal should be called to complete the job. Remember from our discussion of ciphers in Chapter 6 that when a block cipher is being used, the final block of decrypted data will not be written to the output buffer until EVP_DecryptFinal is called (or, in this case, EVP_OpenFinal). Once EVP_OpenFinal is called, the context cannot be reused until EVP_OpenInit is called again.

int EVP_OpenFinal(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl);

OpenSSL provides functions for many types of objects that write the DER representation of the object into a buffer. Each of the functions has a similar signature: the OpenSSL object as the first argument and a buffer as the second. The functions return the number of bytes written to the buffer or, if the buffer is specified as NULL, the number of bytes that would have been written to the buffer is returned. In addition, if a buffer is specified, it is advanced to the byte after the last byte written in order to facilitate writing multiple DER objects to the same buffer.

int i2d_OBJNAME(OBJTYPE *obj, unsigned char **pp);

You’ll note that the second parameter is specified as a pointer to the buffer. This is done so that the pointer can be advanced after the data is written to it. Example 8-2 demonstrates how these functions can be used by writing an RSA public key into a dynamically allocated buffer. The function returns the buffer that was allocated to hold the DER-encoded key and stores the length of the buffer in the second argument.

unsigned char *DER_encode_RSA_public(RSA *rsa, int *len)
{
    unsigned char *buf, *next;
 
    *len = i2d_RSAPublicKey(rsa, NULL);
    buf = next = (unsigned char *)malloc(*len);
    i2d_RSAPublicKey(rsa, &next);
    return buf;
}

Likewise, a function is provided for each type of object that reads the DER representation of the object from a buffer and creates the appropriate object in memory. Again, each of the functions has a similar signature. The first argument is an object to populate with the data obtained from the buffer, which is specified as the second argument. The third argument specifies the number of bytes contained in the buffer that should be used for constructing the object.

               OBJTYPE *d2i_OBJNAME(OBJTYPE **obj, unsigned char **pp, long length);

If the first argument is specified as NULL, a new object of the appropriate type is created and populated with the data recovered from the buffer. If it is specified as a pointer to NULL, a new object of the appropriate type is created and populated in the same manner, but the argument is updated to receive the newly created object. Finally, if a pointer to an existing object is specified, the existing object is populated with the data recovered from the buffer. In all cases, the return value of the function is the object that was populated, or NULL if an error occurred in recovering the data. Example 8-3 demonstrates DER-decoding an RSA public key.

RSA *DER_decode_RSA_public(unsigned char *buf, long len)
{
    RSA *rsa;
 
    rsa = d2i_RSAPublicKey(NULL, &buf, len);
    return rsa;
}

The two functions d2i_PublicKey and d2i_PrivateKey have a different signature from the others. The first argument to these two functions is an integer that specifies the type of key (DH, DSA, or RSA), which is encoded in the buffer. One of the constants EVP_PKEY_DH, EVP_PKEY_DSA, or EVP_PKEY_RSA must be specified to indicate the type of key to expect. The rest of the arguments to these two functions are the same, except they are shifted to make room for the argument that specifies the type of key. The function d2i_AutoPrivateKey is provided to allow OpenSSL to attempt to guess the type of private key that is stored in the buffer. Table 8-1 lists some DER-encoding functions.

The two classes of functions above are useful for reading and writing structures to flat memory, but they still require us to write them to or read them from either a file or BIO. To help with this task, OpenSSL provides functions and macros that handle writing to files and streams. Each of the function names listed in Table 8-1 can be used as a base for building a new name to read or write DER encodings to or from a file or BIO. By appending _bio or _fp to the names of the functions, names for functions for writing to or reading from a BIO or file can be made. For example, the function i2d_DSAparams_bio will write the DER representation of DSA parameters to the specified BIO. The definitions for the BIO and file interface functions are in the header file openssl/x509.h.

There are three exceptions to this rule. The first is that there is no BIO or file equivalent function for i2d_PublicKey or d2i_PublicKey. The second exception is that there is no equivalently named function for the d2i_AutoPrivateKey function. The third exception is that d2i_PrivateKey_bio and d2i_PrivateKey_fp both behave as though they were named d2i_AutoPrivateKey_bio and d2i_AutoPrivateKey_fp.

For the functions that write DER representations to a BIO or a file, the first argument is either a BIO or FILE object, depending on the function that is used. The second argument is the object that will be DER-encoded and written to the BIO or file. The return value from these functions is zero if an error occurs; otherwise, it is nonzero to indicate success.

The functions that read DER representations also require a BIO or FILE object as their first argument, depending on which function is used. The second argument is a pointer to an object of the type that is being read, which is treated just the same as the first argument to the base functions. That is, when NULL or a pointer to NULL is specified, a new object of the appropriate type is created; otherwise, the specified object is populated. The return value from these functions is NULL if an error occurs; otherwise, the return is the object that was populated. Example 8-4 demonstrates.

BIO *bio = BIO_new(BIO_s_memory(  ));
RSA *rsa = RSA_generate_key(1024, RSA_F4, NULL, NULL);
i2d_RSAPrivateKey_bio(bio, rsa);
 
FILE *fp = fopen("rsakey.der", "rb");
RSA *rsa = NULL;
d2i_RSAPrivateKey_fp(fp, &rsa);

The same objects that can be read and written in DER format can also be read and written in PEM format. The interface to the PEM format is somewhat different from the DER interface. To begin with, it supports writing to and reading from a BIO or a file; it does not support memory buffers like the DER interface does. As it turns out, this isn’t much of a limitation since you can just use a memory BIO. All of the function declarations can be found in the header file openssl/pem.h.

The functions for writing public keys and parameters all share the same basic signature. The functions for writing to a BIO require a BIO object as the first argument and the object type as the second argument. The functions for writing to a file require a FILE object as the first argument and the object type as the second argument. Each of the functions, regardless of whether they’re writing to a BIO object or a FILE object, return zero if an error occurs and nonzero if the operation was successful.

The functions for writing private keys are a bit more complex because the PEM format allows them to be encrypted before they’re encoded and written out. Each function requires a BIO or FILE object to write to, the object to be written, a password callback function, and symmetric cipher information.

int PEM_write_OBJNAME(FILE *fp, OBJTYPE *obj, const EVP_CIPHER *enc,
                      unsigned char *kstr, int klen, pem_password_cb callback,
                      void *cb_arg);

The functions that are used to write to a BIO object have the same signature, except that the FILE object is replaced with a BIO object. The return values from functions that write private keys are the same as the values from functions that write public keys and parameters: zero if an error occurs, nonzero otherwise. The password callback function, if one is used, has the following signature:

typedef int (*pem_password_cb)(char *buf, int len, int rwflag, void *cb_arg);

The functions for reading public keys, private keys, and parameters all share a similar signature. Each requires a BIO or a FILE object to read from, an object to populate with the data that was read, and a password callback function.

               OBJTYPE *PEM_read_OBJNAME(FILE *fp, OBJTYPE **obj, pem_password_cb callback,
                           void *cb_arg);

The functions that are used to read from a BIO have the same signature, except that the FILE object is replaced with a BIO object. The return value from the reading functions is always a pointer to the object that was populated with the data that was read. If an error occurs reading the PEM data, NULL is returned. See Table 8-2.