Attack Transferability

The phenomenon of attack transferability was discovered by researchers who found that adversarial samples (drawn from adversarial space) that are specifically designed to cause a misclassification in one model are also likely to cause misclassifications in other independently trained models^10,11—even when the two models are backed by distinctly different algorithms or infrastructures.¹² It is far from obvious why this should be the case, given that, for example, the function that a support vector machine fits to a training data distribution presumably bears little resemblance to the function fit by a deep neural network. Put in a different way, the adversarial spaces in the data manifold of trained machine learning model A have been found to overlap significantly with the adversarial spaces of an arbitrary model B.

Transferability of adversarial attacks has important consequences for practical attacks on machine learning because model parameters are not commonly exposed to users interacting with a system. Researchers have developed practical adversarial evasion attacks on so-called black-box models; i.e., classifiers for which almost no information about the machine learning technique or model used is known.¹³ With access only to test samples and results from the black-box classifier, we can generate a labeled training dataset with which we can train a local substitute model. We then can analyze this local substitute model offline to search for samples that belong to adversarial space. Subsequently, attack transferability allows us to use these adversarial samples to fool the remote black-box model.

Attack transferability is an active area of research,¹⁴ and this work will continue to influence the field of adversarial machine learning in the foreseeable future.

Example: Binary Classifier Poisoning Attack

To concretely illustrate poisoning attacks, let’s demonstrate exactly how the decision boundary of a simple machine learning classifier can be manipulated by an attacker with unbounded query access to system predictions.³⁰ We begin by creating a random synthetic dataset using the sklearn.datasets.make_classification() utility:

from sklearn.datasets import make_classification

X, y = make_classification(n_samples=200,
    n_features=2,
    n_informative=2,
    n_redundant=0,
    weights=[.5, .5],
    random_state=17)

This code is a simple two-feature dataset with 200 samples, out of which we will use the first 100 samples to train the classifier and the next 100 to visually demonstrate that the classifier is appropriately fitted.

For our example, we fit a multilayer perceptron (MLP) classifier to this dataset.³¹ MLPs are a class of simple feed-forward neural networks that can create nonlinear decision boundaries. We import the sklearn.neural_network.MLPClassifier class and fit the model to our dataset:

from sklearn.neural_network import MLPClassifier

clf = MLPClassifier(max_iter=600, random_state=123).fit(X[:100], y[:100])

To inspect what’s going on under the hood, let’s generate a visualization of the classifier’s decision function. We create a two-dimensional mesh grid of points in our input space (X and y values between −3 and 3 with intervals of .01 between each adjacent point) and then extract prediction probabilities for each of the points in this mesh:

import numpy as np

xx, yy = np.mgrid[-3:3:.01, −3:3:.01]
grid = np.c_[xx.ravel(), yy.ravel()]
probs = clf.predict_proba(grid)[:, 1].reshape(xx.shape)

Then we generate a contour plot from this information and overlay the test set on the plot, displaying X₁ on the vertical axis and X₀ on the horizontal axis:

import matplotlib.pyplot as plt

f, ax = plt.subplots(figsize=(12, 9))

# Plot the contour background
contour = ax.contourf(xx, yy, probs, 25, cmap="RdBu",
                      vmin=0, vmax=1)
ax_c = f.colorbar(contour)
ax_c.set_label("$P(y = 1)$")
ax_c.set_ticks([0, .25, .5, .75, 1])

# Plot the test set (latter half of X and y)
ax.scatter(X[100:,0], X[100:, 1], c=y[100:], s=50,
           cmap="RdBu", vmin=-.2, vmax=1.2,
           edgecolor="white", linewidth=1)

ax.set(aspect="equal",
       xlim=(-3, 3), ylim=(-3, 3))

Figure 8-4 shows the result.

Figure 8-4. Decision function contour plot of MLP classifier fitted to our dataset

Figure 8-4 shows that the MLP’s decision function seems to fit quite well to the test set. We use the confidence threshold of 0.5 as our decision boundary. That is, if the classifier predicts that P(y = 1) > 0.5, the prediction is y = 1; otherwise, the prediction is y = 0. We define a utility function plot_decision_boundary() for plotting this decision boundary along with the same test set:³²

    plot_decision_boundary(X, y, probs)

Figure 8-5 shows the result.

Figure 8-5. Decision boundary of MLP classifier fitted to our dataset

We then generate five carefully selected chaff points, amounting to just 5% of the training dataset. We assign the label y = 1 to these points because that is what the classifier would predict (given the current decision function):

num_chaff = 5
chaff_X = np.array([np.linspace(-2, −1, num_chaff),
    np.linspace(0.1, 0.1, num_chaff)]).T
chaff_y = np.ones(num_chaff)

Figure 8-6 illustrates the chaff points (depicted by the star markers), which mostly lie within the y = 1 space (y = 1 is depicted by empty circle markers, and y = 0 is depicted by the filled circle markers).

Figure 8-6. Chaff points depicted in relation to the test set

To mimic online learners that use newly received data points to dynamically and incrementally train the machine learning model, we are going to use scikit-learn’s partial_fit() API for incremental learning that some estimators (including MLPClassifier) implement. We incrementally train our existing classifier by partial-fitting the model to the five new chaff points (attack traffic) that we generated:

clf.partial_fit(chaff_X, chaff_y)

The classifier is now updated with this new malicious information. Now, let’s see how the decision boundary has shifted (see Figure 8-7):

probs_poisoned = clf.predict_proba(grid)[:, 1].reshape(xx.shape)
plot_decision_boundary(X, y, probs, chaff_X, chaff_y, probs_poisoned)

Figure 8-7. Shifted decision boundary after 1x partial fitting of five chaff points

The new decision boundary is the darker of the two curves. Notice that the decision boundary has shifted slightly downward, creating a miniscule gap between the two curves. Any points that lie within this gap would previously have been classified as y = 0, but now would be classified as y = 1. This means that the attacker has been successful in causing a targeted misclassification of samples. Repeating the partial_fit() step iteratively (using the same five chaff points) allows us to observe how much the decision function shifts as the percentage of chaff traffic increases, as demonstrated in Figure 8-8.

A larger-magnitude shift of the decision boundary represents a larger input space of misclassified points, which implies a more serious degradation of model performance.

Figure 8-8. Shifted decision boundaries after 2x, 3x, 4x, and 5x partial fitting of five chaff points (10%, 15%, 20%, 25% attack traffic from upper left to lower right, respectively)

Attacker Knowledge

As you might notice from Figure 8-8, knowledge of the underlying model’s decision function is an important factor in poisoning attacks. How does the attacker know how to select chaff in a way that will cause an effective shift of the decision boundary?

The basic level of access that we assume that any attacker has is the ability to launch an unlimited number of queries to the system and obtain a prediction result. That said, the fewer queries made to a system, the less likely an attacker is to trigger tripwires and cause suspicion. An attacker who has access not only to the prediction result but also to the prediction probabilities is in a much more powerful position because they can then derive decision function gradients (such as those shown in Figure 8-4) that allow for useful optimizations, especially when selecting chaff points on complex decision function surfaces; for example, when the decision function has multiple local minima or maxima.

However, even an attacker without access to the prediction probabilities has access to categorical classification results, which then allows them to infer the decision boundaries of a model by making queries to points around the boundary line (as shown in Figures 8-5 through 8-8). This information is enough for an attacker to select chaff traffic that does not arouse suspicion yet can mislead an online learner.

One scenario in which determining chaff placement requires a bit more reverse engineering of the machine learning system is when the input data is transformed before being fed into a classifier. For instance, if PCA dimensionality reduction is applied to the data, how does an attacker know which dimensions of the input to manipulate? Similarly, if the input goes through some other unknown nonlinear transformation before being fed into a classifier, it is a lot less clear how an attacker should map changes in the raw input to points on the decision surface. As another example, some types of user input cannot be easily modified by a user interacting with the system, such as when classifier input depends on system state or properties that users have no influence over. Finally, determining how much chaff is necessary to cause a meaningful shift of the decision boundary can also be challenging.

The aforementioned challenges can mostly be overcome with enough access to the system, allowing the attacker to extract as much information from a learner as possible. As defenders, the goal is to make it difficult for attackers to make simple inferences about the system that allow them to engage in attacks on the learner without sacrificing the system’s efficacy.

Defense Against Poisoning Attacks

There are some design choices that a machine learning system architect can make that will make it more difficult for even motivated adversaries to poison models. Does the system really need real-time, minute-by-minute online learning capabilities, or can similar value be obtained from a daily scheduled incremental training using the previous day’s data? There are significant benefits to designing online learning systems that behave similarly to an offline batch update system:

• Having longer periods between retraining gives systems a chance to inspect the data being fed into the model.

• Analyzing a longer period of data gives systems a better chance of detecting boiling frog attacks, where chaff is injected gradually over longer periods of time. Aggregating a week’s worth of data, as opposed to the past five minutes of data, allows you to detect chaff being injected at a low rate.

• Attackers thrive on short feedback loops. A detectable shift in the decision boundary as a result of their attack traffic gives them quick positive reinforcement and allows them to continue iterating on their method. Having a week-long update cycle means that attackers will not know if their attack attempt will have the positive outcome they are looking for until the following week.

If you have a mechanism for inspecting incremental training data before feeding it into a partial learning algorithm, there are several ways you can detect poisoning attack attempts:

• Identifying abnormal pockets of traffic that originate from a single IP address or autonomous system (ASN), or have unusual common characteristics; for example, many requests coming in with an abnormal user agent string. You can remove such traffic from the automatic incremental learning mechanism and have an analyst inspect the traffic for signs of attacks.

• Maintaining a calibration set of handcrafted “normal traffic” test data that you run against the model after each period of retraining. If the classification results differ dramatically between previous cycles and the current cycle, there might be tampering involved.

• Defining a threshold around the decision boundary and continuously measuring what percentage of test data points observed fall in that space. For example, suppose that you have a simple linear decision boundary at P(y = 1) = 0.5. If you define the decision boundary threshold region to be 0.4 < P(y = 1) < 0.6, you can count the percentage of test data points observed daily that fall within this region of prediction confidence. If, say, 30% of points fall in this region on average, and you suddenly have 80% of points falling into this region over the past week, this anomaly might signify a poisoning attack—attackers will try to stick as close to the decision boundary as possible to maximize the chances of shifting the boundary without raising alarms.

Because of the large variety of model poisoning attacks that can be launched on a machine learning system, there is no hard-and-fast rule for how to definitively secure a system on this front. This field is an active area of research, and there continue to be algorithmic developments in statistical learning techniques that are less susceptible to poisoning attacks. Robust statistics is frequently cited as a potential solution to make algorithms more resilient to malicious tampering.^{33,34,35,36,37}

Example: Binary Classifier Evasion Attack

Let’s demonstrate the principles of evasion attacks by attempting to find an adversarial example using a rudimentary gradient ascent algorithm. Assuming a perfect-knowledge attacker with full access to the trained machine learning model:

1. We begin with an arbitrarily chosen sample and have the model generate prediction probabilities for it.

2. We dissect the model to find features that are the most strongly weighted in the direction that we want the misclassification to occur; that is, we find a feature J that causes the classifier to be less confident in its original prediction.

3. We iteratively increase the magnitude of the feature until the prediction probability crosses the confidence threshold (typically 0.5).

The pretrained machine learning model that we will be working on is a simple web application firewall (WAF).⁴³ This WAF is a dedicated XSS classifier. Given a string, the WAF will predict whether the string is an instance of XSS. Give it a spin!

Even though real-world attackers will have less than perfect knowledge, assuming a perfect-knowledge attacker enables us to perform a worst-case evaluation of model vulnerabilities and demonstrate some upper bounds on these attacks. In this case, we assume the attacker has access to a serialized scikit-learn Pipeline object and can inspect each stage in the model pipeline.

First, we load the trained model and see what steps the pipeline contains using the Python built-in vars() function:⁴⁴

import pickle

p = pickle.load(open('waf/trained_waf_model'))
vars(p)

> {'steps': [
        ('vectorizer',
         TfidfVectorizer(analyzer='char', binary=False,
                         decode_error=u'strict',
                         dtype=<type 'numpy.int64'>,
                         encoding=u'utf-8', input=u'content',
                         lowercase=True, max_df=1.0,
                         max_features=None, min_df=0.0,
                         ngram_range=(1, 3), norm=u'l2',
                         preprocessor=None, smooth_idf=True,
                         stop_words=None, strip_accents=None,
                         sublinear_tf=True,
                         token_pattern=u'(?u)\\b\\w\\w+\\b',
                         tokenizer=None, use_idf=True,
                         vocabulary=None)
        ),
        ('classifier',
         LogisticRegression(C=1.0, class_weight='balanced',
                            dual=False, fit_intercept=True,
                            intercept_scaling=1, max_iter=100,
                            multi_class='ovr', n_jobs=1,
                            penalty='l2', random_state=None,
                            solver='liblinear', tol=0.0001,
                            verbose=0, warm_start=False)
        )]}

We see that the Pipeline object contains just two steps: a TfidfVectorizer followed by a LogisticRegression classifier. A successful adversarial example, in this case, should cause a false negative; specifically, it should be a string that is a valid XSS payload but is classified as a benign string by the classifier.

Given our previous knowledge of text vectorizers, we know that we now need to find the particular string tokens that can help influence the classifier the most. We can inspect the vectorizer’s token vocabulary by inspecting its vocabulary_ attribute:

vec = p.steps[0][1]
vec.vocabulary_

> {u'\x00\x02': 7,
   u'\x00': 0,
   u'\x00\x00': 1,
   u'\x00\x00\x00': 2,
   u'\x00\x00\x02': 3,
   u'q-1': 73854,
   u'q-0': 73853,
   ...
  }

Each of these tokens is associated with a term weight (the learned inverse document frequency, or IDF, vector) that is fed into the classifier as a single document’s feature. The trained LogisticRegression classifier has coefficients that can be accessed through its coef_ attribute. Let’s inspect these two arrays and see how we can make sense of them:

clf = p.steps[1][1]

print(vec.idf_)

> [  9.88191796  13.29416517  13.98731235  ...,
    14.39277746  14.39277746  14.39277746]

print(clf.coef_)

> [[  3.86345441e+00   2.97867212e-02   1.67598454e-03 ...,
      5.48339628e-06   5.48339628e-06   5.48339628e-06]]

The product of the IDF term weights and the LogisticRegression coefficients determines exactly how much influence each term has on the overall prediction probabilities:

term_influence = vec.idf_ * clf.coef_
print(term_influence)

> [[  3.81783395e+01   3.95989592e-01   2.34425193e-02 ...,
      7.89213024e-05   7.89213024e-05   7.89213024e-05]]

We now want to rank the terms by the value of influence. We use the function numpy.argpartition() to sort the array and convert the values into indices of the vec.idf_ array so that we can find the corresponding token string from the vectorizer’s token dictionary, vec.vocabulary_:

print(np.argpartition(term_influence, 1))

> [[81937 92199     2 ..., 97829 97830 97831]]

It looks like the token at index 80832 has the most positive influence on the prediction confidence. Let’s inspect this token string by extracting it from the token dictionary:

# First, we create a token vocabulary dictionary so that
# we can access tokens by index
vocab = dict([(v,k) for k,v in vec.vocabulary_.items()])

# Then, we can inspect the token at index 80832
print(vocab[81937])

> t/s

Appending this token to an XSS input payload should cause the classifier to be slightly less confident in its prediction. Let’s pick an arbitrary payload and verify that the classifier does indeed correctly classify it as an XSS string (y = 1):

payload = "<script>alert(1)</script>"

p.predict([payload])[0]

# The classifier correctly predicts that this is an XSS payload
> 1

p.predict_proba([payload])[0]

# The classifier is 99.9999997% confident of this prediction
> array([  1.86163618e-09,   9.99999998e-01])

Then, let’s see how appending the string "t/s" to the input affects the prediction probability:

p.predict_proba([payload + '/' + vocab[80832]])[0]

> array([  1.83734699e-07,   9.99999816e-01])

The prediction confidence went down from 99.9999998% to 99.9999816%! All we need to do now is to increase the weight of this feature in this sample. For the TfidfVectorizer, this simply means increasing the number of times this token appears in the input string. As we continue to increase the weight of this feature in the sample, we are ascending the gradient of the classifier’s confidence in the target class; that is, the classifier is more and more confident that the sample is not XSS.

Eventually, we get to the point where the prediction probability for class y = 0 surpasses that for y = 1:

p.predict_proba([payload + '/' + vocab[80832]*258])[0]

> array([ 0.50142443,  0.49857557])

And the classifier predicts that this input string is not an XSS string:

p.predict([payload + '/' + vocab[80832]*258])[0]

> 0

Inspecting the string, we confirm that it is definitely a valid piece of XSS:

print(payload + '/' + vocab[80832]*258)

# Output truncated for brevity
> <script>alert(1)</script>/t/st/st/st/st/st/st/st/st/s...t/s

We have thus successfully found an adversarial sample that fools this machine learning WAF.

The technique we have demonstrated here works for the very simple linear model of this example, but it would be extremely inefficient with even slightly more complex machine learning models. Generating adversarial examples for evasion attacks on arbitrary machine learning models requires more efficient algorithms. There are two predominant methods based on the similar concept of gradient ascent:

Fast Gradient Sign Method (FGSM)⁴⁵

FGSM works by computing the gradient of the classifier’s output with respect to changes in its input. By finding the direction of perturbation that causes the largest change in the classification result, we can uniformly perturb the entire input (i.e., image) by a small amount in that direction. This method is very efficient, but usually requires a larger perturbation to the input than is required to cause a misclassification. For adversarial images, this means that there will appear to be random noise that covers the entire image.

Jacobian Saliency Map Approach (JSMA)⁴⁶

This adversarial sample generation method uses the concept of a saliency map, a map of relative importance for every feature in the input. For images, this map gives a measure of how much a change to the pixel at each position will affect the overall classification result. We can use the salience map to identify a set of the most impactful pixels, and we can then use a gradient ascent approach to iteratively modify as few pixels as possible to cause a misclassification. This method is more computationally intensive than FGSM, but results in adversarial examples that are less likely to be immediately identified by human observers as having been tampered with.

As discussed earlier in this chapter, these attacks can be applied to arbitrary machine learning systems even when attackers have very limited knowledge of the system; in other words, they are black-box attacks.

Defense Against Evasion Attacks

As of this writing, there are no robust defenses against adversarial evasion. The research so far has shown anything that system designers can do to defend against this class of attacks can be overcome by an attacker with more time or computational resources.

Because evasion attacks are driven by the concept of gradient ascent to find samples belonging to adversarial space, the general idea behind defending machine learning models against evasion attacks is to make it more difficult for adversaries to get information about a model’s decision surface gradients. Here are two proposed defense methods:

Adversarial training

If we train our machine learning model with adversarial samples and their correct labels, we may be able to minimize the adversarial space available for attackers to exploit. This defense method attempts to enumerate all possible inputs to a classifier by drawing samples belonging to the theoretical input space that are not covered in the original training data distribution (illustrated in Figure 8-1). By explicitly training models not to be fooled by these adversarial samples, could we perhaps beat attackers at their own game?

Adversarial training has shown promising results, but only solves the problem to a degree since the success of this defense technique rests on winning the arms race between attackers and defenders. Because it is, for most meaningful problem spaces, impossible to exhaustively enumerate the entire theoretical input space, an attacker with enough patience and computational resources can always find adversarial samples on which a model hasn’t explicitly been trained.

Defensive distillation

Distillation was originally designed as a technique for compressing neural network model sizes and computational requirements so that they can run on devices with strict resource limitations such as mobile devices or embedded systems.⁴⁷ This compression is achieved by training an optimized model by replacing the categorical class labels from the original training set with the probability vector outputs of the initial model. The resulting model has a much smoother decision surface that makes it more difficult for attackers to infer a gradient. As with adversarial training, this method only makes it slower and more difficult for attackers to discover and exploit adversarial spaces, and hence solves the problem only against computationally bounded attackers.

Evasion attacks are difficult to defend against precisely because of the issue of imperfect learning⁴⁸—the inability of statistical processes to exhaustively capture all possible inputs that belong to a particular category of items that we would like for classifiers to correctly classify.