In this section, we will start to manually create some features in order to quantify our textual passwords. Let's first create a new column in the data DataFrame called length, which will represent the length of the password:

# 1. the length of the password

# on the left of the equal sign, note we are defining a new column called 'length'. We want this column to hold the 
# length of the password. 

# on the right of the equal sign, we use the apply method of pandas Series/DFs. We will apply a function (len in this case)
# to every element in the column 'text'

data['length'] = data['text'].apply(len)

# see our changes take effect
data.head()

Here is the output:

Let's use this new column to see the most common passwords of five or more characters:

# top passwords of length 5 or more
data[data.length > 4]["text"].value_counts()[:10]

123456       8
123456789    7
12345        6
43162        5
7758521      5
11111        5
5201314      5
111111       4
123321       4
102030       4

These seem more like what we expected; we even see 111111, which was on the list we saw at the beginning of this chapter. We continue now by adding another column, num_caps, that will count the number of capital letters in the password. This will eventually give us some insight into the strength of a password:

# store a new column
data['num_caps'] = data['text'].apply(caps)

# see our changes take effect
data.head(10)

We can now see our two new columns, both of which give us some quantifiable means of assessing password strength. Longer passwords with more capital letters tend to correlate to stronger passwords. But of course this is not the whole picture:

data['num_caps'].hist() # most passwords do not have any caps in them

Running this code will yield the following histogram, revealing a right skew of capital letters, meaning that most people stay on the lower end of capital letters:

Calling the describe method of the DataFrame will reveal some high-level descriptive statistics about our data:

# grab some basic descriptive statistics
data.describe()

Here is the output:

The max row of the length attribute is telling us that we have some massive passwords (over 8,000 characters). We will isolate the passwords that are over 100 characters:

# let's see our long passwords
data[data.length > 100]

The long passwords can be seen here:

We can clearly see that eight of the rows of our DataFrame became malformed. To make this a bit easier, let's use pandas to get rid of these eight problematic rows. We could do work to sanitize this data; however, this case study will focus on deeper insights:

print data[data.length > 100].shape # only 8 rows that became malformed
# to make this easy, let's just drop those problematic rows

# we will drop passwords that are way too long
data.drop(data[data.length > 100].index, axis=0, inplace=True)
(8, 3)


# 1,048,485 - 8 == 1,048,477 makes sense
print data.shape
(1048477, 3)


data.describe()

The following table is the output of the preceding code:

We will now turn to scikit-learn to add some automatic feature extraction.