Avoid Using PDFs!

The data used in this section is the same data as in the previous chapter, but in PDF form. Normally, one does not seek data in difficult-to-parse formats, but we did for this book because the data you need to work with may not always be in the ideal format. You can find the PDF we use in this chapter in the book’s GitHub repository.

There are a few things you need to consider before you start parsing PDF data:

• Have you tried to find the data in another form? If you can’t find it online, try using a phone or email.

• Have you tried to copy and paste the data from the document? Sometimes, you can easily select, copy, and paste data from a PDF into a spreadsheet. This doesn’t always work, though, and it is not scalable (meaning you can’t do it for many files or pages quickly).

If you can’t avoid dealing with PDFs, you’ll need to learn how to parse your data with Python. Let’s get started.

Programmatic Approaches to PDF Parsing

PDFs are more difficult to work with than Excel files because each one can be in an unpredictable format. (When you have a series of PDF files, parsing comes in handy, because hopefully they will be a consistent set of documents.)

PDF tools handle documents in various ways, including by converting the PDFs to text. As we were writing this book, Danielle Cervantes started a conversation about PDF tools on a listserv for journalists called NICAR. The conversation led to the compilation of the following list of PDF parsing tools:

• ABBYY’s Transformer

• Able2ExtractPro

• Acrobat Professional

• Adobe Reader

• Apache Tika

• Cogniview’s PDF to Excel

• CometDocs

• Docsplit

• Nitro Pro

• PDF XChange Viewer

• pdfminer

• pdftk

• pdftotext

• Poppler

• Tabula

• Tesseract

• xPDF

• Zamzar

Besides these tools, you can also parse PDFs with many programming languages—including Python.

As mentioned in “Installing Python Packages”, PyPI is a convenient place for us to look for Python packages. If you search for “PDF”, you’ll get a bunch of results, similar to those shown in Figure 5-1.

Figure 5-1. PDF packages on PyPI

If you start looking through these options to get more information on each of the libraries, none of them will look like an obvious choice for parsing PDFs. If you try a couple more searches, such as “parse pdf,” more options will surface, but there is no obvious leader. So, we went back to search engines to look for what people are using.

After looking at various tutorials, documentation, blog posts, and a couple of helpful articles such as this one, we decided to try the slate library.

pip install slate

import slate                    

pdf = 'EN-FINAL Table 9.pdf'    

with open(pdf, 'rb') as f:      
    doc = slate.PDF(f)          

for page in doc[:2]:            
    print page

TABLE 9Afghanistan 10  11  10  15  40  37  –  –  –  –  90  74  75  74Albania
12  14  9  0  10  99  –  –  –  36  30  75  78  71Algeria 5 y 6 y 4 y 0  2  99
–  –  –  –  68  88  89  87Andorra –   –  –  –  –  100 v –  –  –  –  –  –  –
–Angola 24 x 22 x 25 x –  –  36 x –  –  –  –  –  –  –  –Antigua and Barbuda
–   –   –    –  –  –  –  –  –  –  –  –  –  –Argentina 7 y 8 y 5 y –  –  99 y
–  –  –  –  –  –  –  –Armenia 4  5  3  0  7  100  –  –  –  20  9  70  72
67Australia –   –   –    –  –  100 v –  –  –  –  –  –  –  –Austria –   –
–    –  –  100 v –  –  –  –  –  –  –  –Azerbaijan 7 y 8 y 5 y 1  12  94  –  –
–  58  49  75  79  71Bahamas –   –   –    –  –  –  –  –  –  –  –  –  –
–Bahrain 5 x 6 x 3 x –  –  –  –  –  –  –  –  –  –  –Bangladesh 13    18
8    29  65  31  –  –  –  –  33 y –  –  –Barbados –   –   –    –  –  –  –  –
–  –  –  –  –  –Belarus 1    1    2    0  3  100 y –  –  –  4  4  65 y 67 y 62
yBelgium –   –   –    –  –  100 v –  –  –  –  –  –  –  –Belize 6    7    5
3  26  95  –  –  –  –  9  71  71  70Benin 46    47    45    8  34  80  13  2 y
1  14  47  –  –  –Bhutan 3  3  3  6  26  100  –  –  –  –  68  –  –  –Bolivia (
Plurinational    State of) 26 y 28 y 24 y 3  22  76 y –  –  –  –  16  –  –
–Bosnia and Herzegovina 5    7    4    0  4  100  –  –  –  6  5  55  60
50Botswana 9 y 11 y 7 y –  –  72  –  –  –  –  –  –  –  –Brazil 9 y 11 y 6 y
11  36  93 y –  –  –  –  –  –  –  –Brunei Darussalam –   –   –    –  –  –  –
–  –  –  –  –  –  –Bulgaria –   –   –    –  –  100 v –  –  –  –  –  –  –
–Burkina Faso 39  42  36  10  52  77  76  13  9  34  44  83  84  82Burundi
26    26    27    3  20  75  –  –  –  44  73  –  –  –Cabo Verde 3 x,y 4 x,y 3
x,y 3  18  91  –  –  –  16 y 17  –  –  –Cambodia 36 y 36 y 36 y 2  18  62  –
–  –  22 y 46 y –  –  –Cameroon 42    43    40    13  38  61  1  1 y 7  39
47  93  93  93Canada –   –   –    –  –  100 v –  –  –  –  –  –  –  –Central
African Republic  29    27    30    29  68  61  24  1  11  80 y 80  92  92
92Chad 26    25    28    29  68  16  44  18 y 38  –  62  84  85  84Chile 3 x 3
x 2 x –  –  100 y –  –  –  –  –  –  –  –China –   –   –    –  –  –  –  –  –
–  –  –  –  –Colombia 13 y 17 y 9 y 6  23  97  –  –  –  –  –  –  –  –Comoros
27 x 26 x 28 x –  –  88 x –  –  –  –  –  –  –  –Congo 25    24    25    7  33
91  –  –  –  –  76  –  –  –TABLE 9    CHILD PROTECTIONCountries  and
areasChild labour (%)+ 2005–2012*Child marriage (%) 2005–2012*Birth
registration (%)+ 2005–2012*totalFemale genital mutilation/cutting (%)+
2002–2012*Justification of  wife beating (%) 2005–2012*Violent discipline (%)+
2005–2012*prevalenceattitudestotalmalefemalemarried by 15married by
18womenagirlsbsupport for the practicecmalefemaletotalmalefemale78      THE
STATE OF THE WORLD’S CHILDREN 2014 IN NUMBERS

for page in doc[:2]:
    print type(page)            

<type 'str'>
<type 'str'>

pip install pdfminer

pdf2txt.py -o en-final-table9.txt EN-FINAL\ Table\ 9.pdf

pdf_txt = 'en-final-table9.txt'
openfile = open(pdf_txt, 'r')

for line in openfile:
    print line

Parsing PDFs Using pdfminer

Because PDFs are notoriously difficult to work with, we’ll be learning how to work through problems in our code and do some basic troubleshooting.

We want to start collecting the country names, because the country names are going to be the keys in our final dataset. If you open up your text file, you will find that the eighth line is the last line before the countries start. That line has the text and areas:

5    TABLE 9     CHILD PROTECTION
6
7    Countries
8    and areas
9    Afghanistan
10   Albania
11   Algeria
12   Andorra

If you look through the text document, you will see that this is a consistent pattern. So, we want to create a variable which acts as an on/off switch to start and stop the collection process when it hits the line and areas.

To accomplish this we will update the for loop to include a Boolean variable, which is a True/False variable. We want to set our Boolean to True when we hit the and areas lines:

country_line = False                             
for line in openfile:

    if line.startswith('and areas'):             
        country_line = True                      

Sets country_line to False, because by default the line is not a country line.

Searches for a line that starts with and areas.

Sets country_line to True.

The next thing we need to find is when to set the Boolean back to False. Take a moment to look at the text file to identify the pattern. How do you know when the list of countries ends?

If you look at the follow excerpt, you will notice there is a blank line:

45    China
46    Colombia
47    Comoros
48    Congo
49
50    total
51    10
52    12

But how does Python recognize a blank line? Add a line to your script to print out the Python representation of the line (see “Formatting Data” for more on string formatting):

country_line = False
for line in openfile:
    if country_line:                             
        print '%r' % line                        

    if line.startswith('and areas'):
        country_line = True

If country_line is True, which it will be after the previous iteration of the for loop…

… then print out the Python representation of the line.

If you look at the output, you will notice that all the lines now have extra characters at the end:

45    'China \n'
46    'Colombia \n'
47    'Comoros \n'
48    'Congo \n'
49    '\n'
50    'total\n'
51    '10   \n'
52    '12   \n'

The \n is the symbol of the end of a line, or a newline character. This is what we will now use as the marker to turn off the country_line variable. If country_line is set to True but the line is equal to \n, our code should set country_line to False, because this line marks the end of the country names:

country_line = False
for line in openfile:

    if country_line:                            
        print line

    if line.startswith('and areas'):
        country_line = True
    elif country_line:
        if line == '\n':                        
            country_line = False

If country_line is True, print out the line so we can see the country name. This comes first because we don’t want it after our and areas test; we only want to print the actual country names, not the and areas line.

If country_line is True and the line is equal to a newline character, set country_line to False, because the list of countries has ended.

Now, when we run our code, we get what looks like all the lines with countries returned. This will eventually turn into our country list. Now, let’s look for the markers for the data we want to collect and do the same thing. The data we are looking for is the child labor and child marriage figures. We will begin with child labor data—we need total, male, and female numbers. Let’s start with the total.

We will follow the same pattern to find total child labor:

1. Create an on/off switch using True/False.

2. Look for the starter marker to turn it on.

3. Look for the ending marker to turn it off.

If you look at the text, you will see the starting marker for data is total. Look at line 50 in the text file you created to see the first instance:¹

45    China
46    Colombia
47    Comoros
48    Congo
49
50    total
51    10
52    12

The ending marker is once again a newline or \n, which you can see on line 71:

68    6
69    46
70    3
71
72    26  y
73    5

Let’s add this logic to our code and check the results using print:

country_line = total_line = False               
for line in openfile:

    if country_line or total_line:              
        print line

    if line.startswith('and areas'):
        country_line = True
    elif country_line:
        if line == '\n':
            country_line = False

    if line.startswith('total'):                
        total_line = True
    elif total_line:
        if line == '\n':
            total_line = False

Sets total_line to False.

If country_line or total_line is set to True, outputs the line so we can see what data we have.

Checks where total_line starts and set total_line to True. The code following this line follows the same construction we used for the country_line logic.

At this point, we have some code redundancy. We are repeating some of the same code, just with different variables or on and off switches. This opens up a conversation about how to create non-redundant code. In Python, we can use functions to perform repetitive actions. That is, instead of manually performing those sets of actions line by line in our code, we can put them in a function and call the function to do it for us. If we need to test each line of a PDF, we can use functions instead.

We will name the function we are writing turn_on_off, and we will set it up to receive up to four arguments:

• line is the line we are evaluating.

• status is a Boolean (True or False) representing whether it is on or off.

• start is what we are looking for as the start of the section—this will trigger the on or True status.

• end is what we are looking for as the end of the section—this will trigger the off or False status.

Update your code and add the shell for the function above your for loop. Do not forget to add a description of what the function does—this is so when you refer back to the function later, you don’t have to try to decipher it. These descriptions are called docstrings:

def turn_on_off(line, status, start, end='\n'):                             
    """
        This function checks to see if a line starts/ends with a certain    
        value. If the line starts/ends with that value, the status is
        set to on/off (True/False).
    """
    return status                                                           

country_line = total_line = False
for line in openfile:
    .....

This line begins the function and will take up to four arguments. The first three, line, status, and start, are required arguments—that is, they must be provided because they have no default values. The last one, end, has a default value of a newline character, as that appears to be a pattern with our file. We can override the default value by passing a different value when we call our function.

Always write a description (or docstring) for your function, so you know what it does. It doesn’t have to be perfect. Just make sure you have something. You can always update it later.

The return statement is the proper way to exit a function. In this case, we are going to return status, which will be True or False.

Now, let’s move the code from our for loop into the function. We want to replicate the logic we had with country_line in our new turn_on_off function:

def turn_on_off(line, status, start, end='\n'):
    """
        This function checks to see if a line starts/ends with a certain
        value. If the line starts/ends with that value, the status is
        set to on/off (True/False).
    """

    if line.startswith(start):                  
        status = True
    elif status:
        if line == end:                         
            status = False
    return status                               

Replaces what we are searching for on the starting line with the start variable.

Replaces the end text we used with the end variable.

Returns the status based on the same logic (end means False, start means True).

Let’s now call the function in our for loop, and check out what our script looks like all together thus far:

pdf_txt = 'en-final-table9.txt'
openfile = open(pdf_txt, "r")


def turn_on_off(line, status, start, end='\n'):
    """
        This function checks to see if a line starts/ends with a certain
        value. If the line starts/ends with that value, the status is
        set to on/off (True/False).
    """
    if line.startswith(start):
        status = True
    elif status:
        if line == end:
            status = False
    return status


country_line = total_line = False                                           

for line in openfile:
    if country_line or total_line:                                          
        print '%r' % line

    country_line = turn_on_off(line, country_line, 'and areas')             
    total_line = turn_on_off(line, total_line, 'total')                     

In Python syntax, a series of = symbols means we are assigning each of the variables listed to the final value. This line assigns both country_line and total_line to False.

We want to still keep track of our lines and the data they hold when we are on. For this, we are employing or. A Python or says if one or the other is true, do the following. This line says if either country_line or total_line is True, print the line.

This calls the function for countries. The country_line variable catches the returned status that the function outputs and updates it for the next for loop.

This calls the function for totals. It works the same as the previous line for country names.

Let’s start to store our countries and totals in lists. Then we will take those lists and turn them into a dictionary, where the country will be the key and the total will be the value. This will help us troubleshoot to see if we need to clean up our data.

Here’s the code to create the two lists:

countries = []                                                          
totals = []                                                             
country_line = total_line = False
for line in openfile:
                                                                        
    if country_line:
        countries.append(line)                                          

    elif total_line:
        totals.append(line)                                             

    country_line = turn_on_off(line, country_line, 'and areas')
    total_line = turn_on_off(line, total_line, 'total')

Creates empty countries list.

Creates empty totals list.

Note that we’ve removed the if country_line or total_line statement. We will break this out separately below.

If it’s a country line, this line adds the country to the country list.

This line collects totals, the same as we did with countries.

We are going to combine the totals and countries by “zipping” the two datasets. The zip function takes an item from each list and pairs them together until all items are paired. We can then convert the zipped list to a dictionary by passing it to the dict function.

Add the following to the end of the script:

import pprint                                                           
test_data = dict(zip(countries, totals))                                
pprint.pprint(test_data)                                                

Imports the pprint library. This prints complex data structures in a way that makes them easy to read.

Creates a variable called test_data, which will be the countries and totals zipped together and then turned into a dictionary.

Passes test_data to the pprint.pprint() function to pretty print our data.

If you run the script now, you will get a dictionary that looks like this:

{'\n': '49   \n',
 ' \n': '\xe2\x80\x93   \n',
 '   Republic of Korea \n': '70   \n',
 '   Republic of) \n': '\xe2\x80\x93   \n',
 '   State of) \n': '37  \n',
 '   of the Congo \n': '\xe2\x80\x93   \n',
 '   the Grenadines \n': '60  \n',
 'Afghanistan \n': '10   \n',
 'Albania \n': '12   \n',
 'Algeria \n': '5  y \n',
 'Andorra \n': '\xe2\x80\x93    \n',
 'Angola \n': '24  x \n',
 'Antigua and Barbuda \n': '\xe2\x80\x93    \n',
 'Argentina \n': '7  y \n',
 'Armenia \n': '4   \n',
 'Australia \n': '\xe2\x80\x93    \n',
 ......

At this point, we are going to do some cleaning. This will be explained in greater detail in Chapter 7. For now, we need to clean up our strings, as they are very hard to read. We are going to do this by creating a function to clean up each line. Place this function above the for loop, near your other function:

def clean(line):
    """
        Cleans line breaks, spaces, and special characters from our line.
    """
    line = line.strip('\n').strip()                                     
    line = line.replace('\xe2\x80\x93', '-')                            
    line = line.replace('\xe2\x80\x99', '\'')

    return line                                                         

Strips \n off of the line and reassigns the output to line so now line holds the cleaned version

Replaces special character encodings

Returns our newly cleaned string

line = line.strip('\n').strip().replace(
    '\xe2\x80\x93', '-').replace('\xe2\x80\x99s', '\'')

Let’s apply the clean_line function in our for loop:

for line in openfile:
    if country_line:
        countries.append(clean(line))
    elif total_line:
        totals.append(clean(line))

Now if we run our script, we get something that looks closer to what we are aiming for:

{'Afghanistan': '10',
 'Albania': '12',
 'Algeria': '5  y',
 'Andorra': '-',
 'Angola': '24  x',
 'Antigua and Barbuda': '-',
 'Argentina': '7  y',
 'Armenia': '4',
 'Australia': '-',
 'Austria': '-',
 'Azerbaijan': '7  y',
...

If you skim the list, you will see our approach is not adequately parsing all of the data. We need to figure out why this is happening.

It looks like countries with names spread over more than one line are separated into two records. You can see with this Bolivia: we have records reading 'Bolivia (Plurinational': '', and 'State of)': '26 y',.

The PDF itself can be used as a visual reference to show you how the data should be organized. You can see these lines in the PDF, as shown in Figure 5-2.

Figure 5-2. Bolivia in the PDF

There are a couple of ways to approach this problem. We could try to create a placeholder to check for blank total lines and combine those with the following data lines. Another solution is to keep track of which countries have records spanning more than one line. We will try the second approach, as our dataset isn’t too large.

We will create a list of the first lines for each multiline country record and use this list to check each line in our script. You will want to put this list before your for loop. Often, reference items are put near the top of the script so they are easy to find and change as needed.

Let’s add Bolivia (Plurinational to a list of double-lined countries:

double_lined_countries = [
    'Bolivia (Plurinational',
]

Now we need to update our for loop to check if the previous line is in the double_lined_countries list, and, if so, combine that line with the current line. To do so, we will create a previous_line variable. Then, we will populate the previous_line variable at the end of the for loop. Only then will we be able to combine the rows when the code hits the next iteration of the loop:

countries = []
totals = []
country_line = total_line = False
previous_line = ''                                                        

for line in openfile:
    if country_line:
        countries.append(clean(line))
    elif total_line:
        totals.append(clean(line))

    country_line = turn_on_off(line, country_line, 'and areas')
    total_line = turn_on_off(line, total_line, 'total')

    previous_line = line                                                  

Creates the previous_line variable and sets it to an empty string.

Populates the previous_line variable with the current line at the end of the for loop.

Now we have a previous_line variable and we can check to see if the previous_line is in double_lined_countries so we know when to combine the previous line with the current line. Furthermore, we will want to add the new combined line to the countries list. We also want to make sure we do not add the line to the countries list if the first part of the name is in the double_lined_countries list.

Let’s update our code to reflect these changes:

    if country_line:                                                    
        if previous_line in double_lined_countries:
            line = ' '.join([clean(previous_line), clean(line)])        
            countries.append(line)
        elif line not in double_lined_countries:                        
            countries.append(clean(line))

We want the logic in the if country_line section because it is only relevant to country names.

If the previous_line is in the double_lined_countries list, this line joins the previous_line with the current line and assigns the combined lines to the line variable. join, as you can see, binds a list of strings together with the preceding string. This line uses a space as the joining character.

If the line is not in the double_lined_countries list, then the following line adds it to the countries list. Here, we utilize elif, which is Python’s way of saying else if. This is a nice tool to use if you want to include a different logic flow than if - else.

If we run our script again, we see 'Bolivia (Plurinational State of)' is now combined. Now we need to make sure we have all the countries. We will do this manually because our dataset is small, but if you had a larger dataset, you would automate this.

Look at the PDF in a PDF viewer to identify all the double-lined country names:

Bolivia (Plurinational State of)
Democratic People’s Republic of Korea
Democratic Republic of the Congo
Lao People’s Democratic Republic
Micronesia (Federated States of)
Saint Vincent and the Grenadines
The former Yugoslav Republic of Macedonia
United Republic of Tanzania
Venezuela (Bolivarian Republic of)

We know this is likely not how Python sees it, so we need to print out the countries as Python sees them and add those to the list:

    if country_line:
        print '%r' % line                                               
        if previous_line in double_lined_countries:

Adds a print '%r' statement to output the Python representation

Run your script to populate the double_lined_countries list with the Python representation:

double_lined_countries = [
    'Bolivia (Plurinational \n',
    'Democratic People\xe2\x80\x99s \n',
    'Democratic Republic \n',
    'Micronesia (Federated \n',
    #... uh oh.
]

We are missing Lao People’s Democratic Republic from our output, but it’s on two lines in the PDF. Let’s go back to the text version of the PDF and see what happened.

After looking at the text, can you identify the issue? Look at the turn_on_off function. How does that work in relation to how this text is written?

The problem turns out to be a blank line or \n right after the and areas we were looking for as a marker. If you look at the text file that we created, you will see the stray blank line on line number 1343:

...
1341 Countries
1342 and areas
1343
1344 Iceland
1345 India
1346 Indonesia
1347 Iran (Islamic Republic of)
...

That means our function didn’t work. There are multiple ways we could approach this problem. For this example, we could try adding in more logic to make sure our on/off code works as intended. When we start to collect countries, there should be at least one country collected before we turn off the country collection. If no countries have been collected, then we should not turn off the collecting action. We can also use the previous line as a way to solve this problem. We can test the previous line in our on/off function and ensure it’s not in a list of special lines.

In case we come across any other anomalies, let’s set up this special line:

def turn_on_off(line, status, start, prev_line, end='\n'):
    """
        This function checks to see if a line starts/ends with a certain
        value. If the line starts/ends with that value, the status is
        set to on/off (True/False) as long as the previous line isn't special.
    """
    if line.startswith(start):
        status = True
    elif status:
        if line == end and prev_line != 'and areas': 
            status = False
    return status

If the line is equal to end and the previous line is not equal to and areas, then we can turn data collection off. Here, we are using != which is Python’s way of testing “not equal.” Similar to ==, != returns a Boolean value.

You will also need to update your code to pass the previous line:

    country_line = turn_on_off(line, country_line, previous_line, 'and areas')
    total_line = turn_on_off(line, total_line, previous_line, 'total')

Let’s go back to the original task we were working on—creating our list of double-lined countries so that we can make sure to collect both lines. We left off here:

double_lined_countries = [
    'Bolivia (Plurinational \n',
    'Democratic People\xe2\x80\x99s \n',
    'Democratic Republic \n',
]

Looking at the PDF, we see the next one is Lao People’s Democratic Republic. Let’s start adding from there by looking back at our script output:

double_lined_countries = [
    'Bolivia (Plurinational \n',
    'Democratic People\xe2\x80\x99s \n',
    'Democratic Republic \n',
    'Lao People\xe2\x80\x99s Democratic \n',
    'Micronesia (Federated \n',
    'Saint Vincent and \n',
    'The former Yugoslav \n',
    'United Republic \n',
    'Venezuela (Bolivarian \n',
]

If your list looks like the preceding list, when you run the script, you should have an output that pulls in the country names split over two lines. Make sure to add a print statement to the end of your script to view the country list:

import pprint
pprint.pprint(countries)

Now that you have spent a bit of time with the country list, can you think of another approach to solve this issue? Take a look at a few of the second lines:

'   Republic of Korea \n'
'   Republic \n'
'   of the Congo \n'

What do they have in common? They start with spaces. Writing code to check whether a line begins with three spaces would be more efficient. However, taking the approach we did allowed us to discover we were losing part of our dataset as it was being collected. As your coding skills develop, you will learn to find different ways to approach the same problem and determine which works best.

Let’s check to see how our total numbers line up with our countries. Update the pprint statement to match the following:

import pprint
data = dict(zip(countries, totals))             
pprint.pprint(data)                             

Zips the countries and totals lists together by calling zip(countries, totals). This turns them into a tuple. We then change the tuple into a dictionary, or dict (for easier reading), by passing it to the dict function.

Prints out the data variable we just created.

What you will see returned is a dictionary where the country names are the keys and the totals are the values. This is not our final data format; we are just doing this to see our data so far. The result should look like this:

{'': '-',
'Afghanistan': '10',
 'Albania': '12',
 'Algeria': '5  y',
 'Andorra': '-',
 'Angola': '24  x',
 ...
}

If you check this again alongside the PDF, you will notice it falls apart right at the point of the first country on a double line. The numbers pulled in are the ones from the Birth registration column:

{
 ...
 'Bolivia (Plurinational State of)': '',
 'Bosnia and Herzegovina': '37',
 'Botswana': '99',
 'Brazil': '99',
 ...
}

If you look at the text version of the PDF, you will notice there is a gap between the numbers when the country is on a double line:

6
46
3

26  y
5
9  y

In the same way we accounted for this in the country name collection, we need to account for this in data collection. If we have a blank space in our data lines, we need to make sure we don’t collect it—that way, we only collect the data that matches the countries we’ve been collecting. Update your code to the following:

for line in openfile:
    if country_line:
        print '%r' % line
        if previous_line in double_lined_countries:
            line = ' '.join([clean(previous_line), clean(line)])
            countries.append(line)
        elif line not in double_lined_countries:
            countries.append(clean(line))

    elif total_line:
        if len(line.replace('\n', '').strip()) > 0:             
            totals.append(clean(line))

    country_line = turn_on_off(line, country_line, previous_line,
                                      'and areas')

    total_line = turn_on_off(line, total_line, previous_line,
                                        'total')
    previous_line = line

We know from experience the PDF uses newlines as blank lines. On this line, the code replaces newlines with nothing and strips whitespace to clean it. Then this code tests whether the string still has a length greater than zero. If so, we know there is data and we can add it to our data (totals) list.

After running our updated code, things fall apart again at our first double line. This time we are pulling in those Birth registration numbers again, aligned with our first double-lined country. All the following values are also incorrect. Let’s go back to the text file and figure out what is happening. If you look at the numbers in that column in the PDF, you can find the pattern in the text version of the PDF starting on line number 1251:

1250
1251    total
1252    –
1253    5  x
1254    26
1255    –
1266    –

If you look closely, you will notice the title of the Birth registration column ends in total:

266     Birth
267     registration
268     (%)+
269     2005–2012*
270     total
271     37
272     99

Right now the function collecting totals is looking for total, so this column is getting picked up before we even get to the next line of countries. We also see that the Violent discipline (%) column has a label for total with a blank line above it. This follows the same pattern as the total we want to collect.

Encountering back-to-back bugs likely means the problem exists in the larger logic you’ve constructed. Because we started our script using these on/off switches, to fix the underlying problem we would need to rework the logic there. We’d need to figure out how to best determine the right column, maybe by collecting column names and sorting them. We might need to determine a way to see if the “page” has changed. If we keep quick-fixing the solution, we will likely run into more errors.

This is the process of programming—write code, debug, write code, debug. No matter how experienced a computer programmer you are, there will be moments when you introduce errors in your code. When learning to code, these moments can be demoralizing. You might think, “Why isn’t this working for me? I must not be good at this.” But that’s not true; programming takes practice, like anything else.

At this point, it’s clear our current process isn’t working. Based on what we now know about the text file, we can tell we began with a false notion that the file defines the beginning and end of each section using text. We could begin again with this file, starting at a new point; however, we want to explore some other ways of problem solving to fix the errors and get the data we want.

Learning How to Solve Problems

There are a variety of exercises you can try to parse the PDF script while also challenging your ability to write Python. First, let’s review our code so far:

pdf_txt = 'en-final-table9.txt'
openfile = open(pdf_txt, "r")

double_lined_countries = [
    'Bolivia (Plurinational \n',
    'Democratic People\xe2\x80\x99s \n',
    'Democratic Republic \n',
    'Lao People\xe2\x80\x99s Democratic \n',
    'Micronesia (Federated \n',
    'Saint Vincent and \n',
    'The former Yugoslav \n',
    'United Republic \n',
    'Venezuela (Bolivarian \n',
]


def turn_on_off(line, status, prev_line, start, end='\n', count=0):
    """
        This function checks to see if a line starts/ends with a certain
        value. If the line starts/ends with that value, the status is
        set to on/off (True/False) as long as the previous line isn't special.
    """
    if line.startswith(start):
        status = True
    elif status:
        if line == end and prev_line != 'and areas':
            status = False
    return status


def clean(line):
    """
        Clean line breaks, spaces, and special characters from our line.
    """
    line = line.strip('\n').strip()
    line = line.replace('\xe2\x80\x93', '-')
    line = line.replace('\xe2\x80\x99', '\'')

    return line


countries = []
totals = []
country_line = total_line = False
previous_line = ''


for line in openfile:
    if country_line:
        if previous_line in double_lined_countries:
            line = ' '.join([clean(previous_line), clean(line)])
            countries.append(line)
        elif line not in double_lined_countries:
            countries.append(clean(line))

    elif total_line:
        if len(line.replace('\n', '').strip()) > 0:
            totals.append(clean(line))

    country_line = turn_on_off(line, country_line, previous_line,
                               'and areas')
    total_line = turn_on_off(line, total_line, previous_line,
                             'total')
    previous_line = line


import pprint
data = dict(zip(countries, totals))
pprint.pprint(data)

There are multiple solutions to the problems we are facing; we’ll walk through some of them in the following sections.

from pdftables import get_tables

all_tables = get_tables(open('EN-FINAL Table 9.pdf', 'rb'))

print all_tables

print all_tables[0][:6]

... [u'',
  u'',
  u'',
  u'',
  u'',
  u'',
  u'Birth',
  u'Female',
  u'genital mutila',
  u'tion/cutting (%)+',
  u'Jus',
  u'tification of',
  u'',
  u'',
  u'E'],
 [u'',
  u'',
  u'Child labour (%',
  u')+',
  u'Child m',
  u'arriage (%)',
  u'registration',
  u'',
  u'2002\u201320',
  u'12*',
  u'wife',
  u'beating (%)',
  u'',
  u'Violent disciplin',
  u'e (%)+ 9'],
 [u'Countries  and areas',
  u'total',
  u'2005\u20132012*male',
  u'female',
  u'2005married by 15',
  u'\u20132012*married by 18',
  u'(%)+ 2005\u20132012*total',
  u'prwomena',
  u'evalencegirlsb',
  u'attitudessupport for thepracticec',
  u'2male',
  u'005\u20132012*female',
  u'total',
  u'2005\u20132012*male',
  u'female'],...

headers = ['Country', 'Child Labor 2005-2012 (%) total',
           'Child Labor 2005-2012 (%) male',
           'Child Labor 2005-2012 (%) female',
           'Child Marriage 2005-2012 (%) married by 15',
           'Child Marriage 2005-2012 (%) married by 18',
           'Birth registration 2005-2012 (%)',
           'Female Genital mutilation 2002-2012 (prevalence), women',
           'Female Genital mutilation 2002-2012 (prevalence), girls',
           'Female Genital mutilation 2002-2012 (support)',
           'Justification of wife beating 2005-2012 (%) male',
           'Justification of wife beating 2005-2012 (%) female',
           'Violent discipline 2005-2012 (%) total',
           'Violent discipline 2005-2012 (%) male',
           'Violent discipline 2005-2012 (%) female'] 


for table in all_tables:
    for row in table:
        print zip(headers, row) 

for table in all_tables:
    for row in table[5:]: 
        if row[2] == '': 
            print row

first_name = ''

for table in all_tables:
    for row in table[5:]:
        if row[0] == '': 
            continue
        if row[2] == '':
            first_name = row[0] 
            continue
        if row[0].startswith('    '): 
            row[0] = '{} {}'.format(first_name, row[0])
        print zip(headers, row) 
        if row[0] == 'Zimbabwe':
            break 

[('Country', u'80      THE STATE OF T'),
('Child Labor 2005-2012 (%) total', u'HE WOR'),
('Child Labor 2005-2012 (%) male', u'LD\u2019S CHILDRE'),
('Child Labor 2005-2012 (%) female', u'N 2014'),
('Child Marriage 2005-2012 (%) married by 15', u'IN NUMBER'),
('Child Marriage 2005-2012 (%) married by 18', u'S'),
('Birth registration 2005-2012 (%)', u''),
.....

from pdftables import get_tables
import pprint

headers = ['Country', 'Child Labor 2005-2012 (%) total',
           'Child Labor 2005-2012 (%) male',
           'Child Labor 2005-2012 (%) female',
           'Child Marriage 2005-2012 (%) married by 15',
           'Child Marriage 2005-2012 (%) married by 18',
           'Birth registration 2005-2012 (%)',
           'Female Genital mutilation 2002-2012 (prevalence), women',
           'Female Genital mutilation 2002-2012 (prevalence), girls',
           'Female Genital mutilation 2002-2012 (support)',
           'Justification of wife beating 2005-2012 (%) male',
           'Justification of wife beating 2005-2012 (%) female',
           'Violent discipline 2005-2012 (%) total',
           'Violent discipline 2005-2012 (%) male',
           'Violent discipline 2005-2012 (%) female']

all_tables = get_tables(open('EN-FINAL Table 9.pdf', 'rb'))

first_name = False
final_data = []

for table in all_tables:
    for row in table[5:]:
        if row[0] == '' or row[0][0].isdigit():
            continue
        elif row[2] == '':
            first_name = row[0]
            continue
        if first_name: 
            row[0] = u'{} {}'.format(first_name, row[0])
            first_name = False 
        final_data.append(dict(zip(headers, row)))
        if row[0] == 'Zimbabwe':
            break

pprint.pprint(final_data)

Exercise: Try Another Tool

When we first started to look for a Python library to use for parsing PDFs, we found slate—which looked easy to use but required some custom code—by searching the Web to see what other people were using for the task.

To see what else was out there, instead of searching for “parsing pdfs python,” we tried searching for “extracting tables from pdf,” which gave us more distinct solutions for the table problem (including a blog post reviewing several tools).

With a small PDF like the one we are using, we could try Tabula. Tabula isn’t always going to be the solution, but it has some good capabilities.

To get started with Tabula:

1. Download Tabula.

2. Launch the application by double-clicking on it; this will launch the tool in your browser.

3. Upload the child labor PDF.

From here, you need to adjust the selection of the content Tabula tries to grab. Getting rid of the header rows enables Tabula to identify the data on every page and automatically highlight it for extraction. First, select the tables you are interested in (see Figure 5-3).

Figure 5-3. Select tables in Tabula

Next, download the data (see Figure 5-4).

Figure 5-4. Download screen in Tabula

Click “Download CSV” and you will get something that looks like Figure 5-5.

Figure 5-5. Exacted data in CSV form

It’s not perfect, but the data is cleaner than we received from pdfminer.

The challenge will be to take the CSV you created with Tabula and parse it. It is different from the other CSVs we parsed (in Chapter 3) and a little messier. If you get stumped, put it aside and come back to it after reading Chapter 7.

Uncommon File Types

So far in this book, we have covered CSV, JSON, XML, Excel, and PDF files. Data in PDFs can be difficult to parse, and you may think the world of data wrangling can’t get any worse—sadly, it can.

The good news is, there’s probably no problem you will face that someone hasn’t solved before. Remember, asking the Python or greater open source communities for help and tips is always a great solution, even if you come away learning you should keep looking for more accessible datasets.

You may encounter problems if the data has the following attributes:

• The file was generated on an old system using a an uncommon file type.

• The file was generated by a proprietary system.

• The file isn’t launchable in a program you have.

Solving problems related to uncommon file types is simply a matter of building on what you have already learned:

1. Identify the file type. If this is not easily done through the file extension, then use the python-magic library.

2. Search the Internet for “how to parse <file extension> in Python,” replacing “<file extension>” with the actual file extension.

3. If there is no obvious solution, try opening the file in a text editor or reading the file with Python’s open function.

4. If the characters look funky, read up on Python encoding. If you are just getting started with Python character encoding, then watch the PyCon 2014 talk “Character encoding and Unicode in Python”.

Summary

PDF and other hard-to-parse formats are the worst formats you will encounter. When you find data in one of these formats, the first thing you should do is see if you can acquire the data in another format. With our example, the data we received in the CSV format was more precise because the numbers were rounded for the PDF charts. The more raw the format, the more likely it is to be accurate and easy to parse with code.

If it is not possible to get the data in another format, then you should try the following process:

1. Identify the data type.

2. Search the Internet to see how other folks have approached the problem. Is there a tool to help import the data?

3. Select the tool that is most intuitive to you. If it’s Python, then select the library that makes the most sense to you.

4. Try to convert the data to a an easier to use format.

In this chapter, we learned about the libraries and tools in Table 5-1.

Besides learning about new tools, we also learned a few new Python programming concepts, which are summarized in Table 5-2.

In the next chapter, we’ll talk about data acquisition and storage. This will provide more insight on how to acquire alternative data formats. In Chapters 7 and 8, we cover data cleaning, which will also help in the complexity of processing PDFs.