Local files

When using local files for input and output, you’ll want to ensure the script can run on the same machine every day, or can be easily moved with the input and output files. As your script grows, it’s possible you will move and change it along with the files you use.

We’ve used local files before, but let’s review how to do so from a more functional code standpoint. This code gives you the ability to open and write files using standard data types, and is very reusable and expandable depending on your script’s needs:

from csv import reader, writer


def read_local_file(file_name):
    if '.csv' in file_name: 
        rdr = reader(open(file_name, 'rb'))
        return rdr
    return open(file_name, 'rb') 


def write_local_file(file_name, data):
    with open(file_name, 'wb') as open_file: 
        if type(data) is list: 
            wr = writer(open_file)
            for line in data:
                wr.writerow(line)
        else:
            open_file.write(data) 

This line tests whether the file might be a good candidate to open with the csv module. If it ends in .csv, then it’s likely we might want to open it using our CSV reader.

If we haven’t returned with our CSV reader, this code returns the open file. If we wanted to build a series of different ways to open and parse files based on the file extension, we could do that as well (e.g., using the json module for JSON files, or pdfminer for PDFs).

This code uses with...as to return the output of the open function, assigning it to the open_file variable. When the indented block ends, Python will close the file automatically.

If we are dealing with a list, this line uses the CSV writer to write each list item as a row of data. If we have dictionaries, we might want to use the DictWriter class.

We want a good backup plan in case it’s not a list. For this reason, write the raw data to file. Instead of this option, we could write different code depending on the data type.

Let’s look at an example where we need the most recent file in a directory, which is often useful if you need to parse log files going back in time or look at the results of a recent web spider run:

import os

def get_latest(folder):
    files = [os.path.join(folder, f) for f in os.listdir(folder)] 
    files.sort(key=lambda x: os.path.getmtime(x), reverse=True) 
    return files[0] 

Uses Python’s built-in os module to list each file (listdir method), then uses the path module’s join method to make a long string representing a full file path. This is an easy way to get a list of all of the files in a folder just by passing a string (the folder’s path).

Sorts files by last-modified date. Because files is a list, we can call the sort method and give it a key on which to sort. This code passes the full file paths to getmtime, which is the os module’s “get modified time” method. The reverse argument makes sure the more recent files are on the top of the list.

Returns only the most recent file.

This code returns the most recent folder, but if we wanted to return the whole list of files starting with the most recent we could simply modify the code to not return the first index, but instead the whole list or a slice.

Config files

Setting up local config files for your sensitive information is a must. As asserted in the Twelve-Factor App, storing your configuration (such as passwords, logins, email addresses, and other sensitive information) outside of your code base is part of being a good developer. If you connect to a database, send an email, use an API, or store payment information, that sensitive data should be stored in a configuration file.

Usually, we store config files in a separate folder within the repository (e.g., config/). All the code in the repository has access to these files, but by using .gitignore files, we can keep the configuration out of version control. If you have other developers or servers who need those files, you should copy them over manually.

Using a folder rather than one file allows you to have different configurations depending on what machine or environment the script runs in. You might want to have one configuration file for the test environment with test API keys, and a production file. You might have more than one database depending on what machine the script uses. You can store these specific pieces of information using a .cfg file, like the following example:

# Example configuration file
[address] 
name = foo 
email = myemail@bar.com
postalcode = 10177
street = Schlangestr. 4
city = Berlin
telephone = 015745738292950383

[auth_login]
user = test@mysite.com
pass = goodpassword

[db]
name = my_awesome_db
user = script_user
password = 7CH+89053FJKwjker)
host = my.host.io

[email]
user = script.email@gmail.com
password = 788Fksjelwi&

Each section is denoted by square brackets with an easy-to-read string inside of them.

Each line contains a key = value pair. The ConfigParser interprets these as strings. Values can contain any characters, including special characters, but keys should follow PEP-8 easy-to-read syntax and structure.

Having sections, keys, and values for our configuration lets us use the names of the sections and keys to access configuration values. This adds clarity to our Python scripts, without being insecure. Once you have a config file like the previous example set up, it’s quite easy to parse with Python and use in your script and automation. Here’s an example:

import ConfigParser
from some_api import get_client 


def get_config(env):
    config = ConfigParser.ConfigParser() 
    if env == 'PROD':
        return config.read(['config/production.cfg']) 
    elif env == 'TEST':
        return config.read(['config/test.cfg'])
    return config.read(['config/development.cfg']) 


def api_login():
    config = get_config('PROD') 
    my_client = get_client(config.get('api_login', 'user'),
                           config.get('api_login', 'auth_key')) 
    return my_client

Here’s an example of an API client hook we could import.

This code instantiates a config object by calling the ConfigParser class. This is now an empty configuration object.

This line calls the configuration parser object’s read method and passes a list of configuration files. Here, we store them in a directory in the root of the project in a folder called config.

If the environment variable passed does not match production or testing, we will always return the development configuration. It’s a good idea to have catches like this in your configuration code, in case of a failure to define environment variables.

We’ll assume our example needs the production API, so this line asks for the PROD configuration. You can also save those types of decisions in the bash environment and read them using the built-in os.environ method.

This line calls the section name and key name to access the values stored in the configuration. This will return the values as strings, so if you need integers or other types, you should convert them.

The built-in ConfigParser library gives us easy access to our sections, keys, and values stored in our config file. If you’d like to store different pieces of information in different files and parse a list of them for each particular script, your code might look like this:

config = ConfigParser.ConfigParser()
config.read(['config/email.cfg', 'config/database.cfg', 'config/staging.cfg'])

It’s up to you to organize your code and configuration depending on your needs. The syntax to access the configuration values simply uses the section name in your config (i.e., [section_name]) and the name of the key. So, a config file like this one:

[email]
user = test@mydomain.org
pass = my_super_password

can be accessed like this:

email_addy = config.get('email', 'user')
email_pass = config.get('email', 'pass')

Command-line arguments

Python gives us the ability to pass command-line arguments to use for automation. These arguments pass information regarding how the script should function. For example, if we need the script to know we want it to run with the development configuration, we could run it like so:

python my_script.py DEV

We are using the same syntax to run a file from the command line, calling python, then the script name, and then adding DEV to the end of the line. How can we parse the extra argument using Python? Let’s write code that does just that:

from import_config import get_config
import sys


def main(env):
    config = get_config(env)
    print config


if __name__ == '__main__':
    if len(sys.argv) > 1: 
        env = sys.argv(1) 
    else:
        env = 'TEST'
    main(env) 

The built-in sys module helps with system tasks, including parsing command-line arguments. If the command-line argument list returned has a length greater than 1, there are extra arguments. The first argument always holds the name of the script (so if it has a length of 1, that’s the only argument).

To get the value of an argument, pass the index of that argument to the sys module’s argv method. This line sets env equal to that value. Remember, the 0-index of argv will always be the Python script name, so you start parsing with the argument at the 1-index.

This line uses the parsed arguments to modify your code according to the command-line arguments.

Argument variables are great if you need to reuse the same code to perform different tasks or run in different environments. Maybe you have a script to run either collection or analysis, and you’d like to switch which environments you use. You might run it like so:

python my_script.py DEV ANALYSIS
python my_script.py PROD COLLECTION

Or you might have a script that needs to interact with a newly updated file folder and grab the latest logs—for example, to grab logs from more than one place:

python my_script.py DEV /var/log/apache2/
python my_script.py PROD /var/log/nginx/

With command-line arguments, simple changes in argument variables can create a portable and robust automation. Not every script will need to use these types of extra variables, but it’s a nice-to-have solution built into the standard Python library and provides some flexibility should you need it.

Aside from these fairly simple and straightforward ways to parse your data and to give your script extra pieces of information, you can use more sophisticated and distributed approaches like cloud data and databasing. We’ll look at these next.

Using Git to deploy Python

If you’d like to have your automation run somewhere other than your local machine, you’ll need to deploy your Python script. We will review a few simple ways to do so, and then some slightly more complex ways.

The easiest and most intuitive way to deploy Python is to put your repository under version control using Git and use Git deploy hooks to “ship” code to your remote hosts. First, you’ll need to install Git.

If you’re new to Git, we recommend taking the Code School tutorial on GitHub or walking through the Git tutorials on Atlassian. It’s fairly easy to get started, and you’ll get the hang of the most used commands quickly. If you’re working on the repository by yourself, you won’t have to worry too much about pulling remote changes, but it’s always good to set a clear routine.

Once your Git installation is complete, run these commands in your project’s code folder:

git init . 
git add my_script.py 
git commit -a 

Initializes the current working directory as the root of your Git repository.

Adds my_script.py to the repository. Use a filename or folder from your repository—just not your config files!

Commits those changes along with any other running changes (-a) to your repository.

When prompted, you will need to write a commit message giving a brief explanation of the changes you’ve made, which should be explicit and clear. You might later need to find which commits implemented certain changes in your code. If you always write clear messages, this will help you search for and find those commits. It will also help others on your team or coworkers understand your code and commits.

It’s also essential you set up a .gitignore file, where you list all of the file patterns you want Git to ignore when you push/pull changes, as discussed in the sidebar “Git and .gitignore”. You can have one for each folder or just one in the base folder of the repository. Most Python .gitignore files look something like this:

*.pyc
*.csv
*.log
config/*

This file will prevent compiled Python files, CSV files, log files, and config files from being stored in the repository. You’ll probably want to add more patterns, depending on what other types of files you have in your repository folders.

You can host your repository on a number of sites. GitHub offers free public repositories but no private repositories. If you need your code to be private, Bitbucket has free private repositories. If you’ve already started using Git locally, it’s easy to push your existing Git repository to GitHub or Bitbucket.

Once you have your repository set up, setting up your remote endpoints (server or servers) with Git is simple. Here is one example if you are deploying to a folder you have ssh access to:

git remote add deploy ssh://user@342.165.22.33/home/user/my_script

Before you can push your code to the server, you’ll need to set up the folder on the receiving end with a few commands. You will want to run these commands in the server folder in which you plan to deploy:

git init .
git config core.worktree `pwd`
git config receive.denycurrentbranch ignore

Here you have initialized an empty repository to send code to from your local machine and defined some simple configurations so Git knows it will be a remote endpoint. You’ll also want to set up a post-receive hook. Do so by creating an executable (via permissions) file called post-receive in the .git/hooks folder in the folder you just initialized. This file will execute when the deploy endpoint receives any Git push. It should contain any tasks you need to run every time you push, such as syncing databases, clearing the cache, or restarting any processes. At a minimum, it will need to update the endpoint.

A simple .git/hooks/post-receive file looks like this:

#!/bin/sh
git checkout -f
git reset --hard

This will reset any local changes (on the remote machine) and update the code.

Once your endpoint is set up, you can simply run the following command from your local repository to update the code on the server with all the latest commits:

git push deploy master

Doing so is a great way to manage your repository and server or remote machine; it’s really easy to use and set up and makes migration, if necessary, straightforward.

If you’re new to deployment and version control, we recommend starting with Git and getting comfortable with it before moving on to more complex deployment options, like using Fabric. Later in this chapter, we’ll cover some larger-scale automation for deploying and managing code across multiple servers.

Email

Emailing with Python is straightforward. We recommend setting up a separate script-only email address through your favorite email provider (we used Gmail). If it doesn’t automatically integrate with Python out of the box, it’s likely there is a listing of the proper configuration or a useful example configuration online, found via search.

Let’s take a look at a script we’ve used to send mail with attachments to a list of recipients. We modified this code from a gist written by @dbieber, which was modified from Rodrigo Coutinho’s “Sending emails via Gmail with Python” post:

#!/usr/bin/python
# Adapted from
# http://kutuma.blogspot.com/2007/08/sending-emails-via-gmail-with-python.html
# Modified again from: https://gist.github.com/dbieber/5146518
# config file(s) should contain section 'email' and parameters
# 'user' and 'password'

import smtplib 
from email.MIMEMultipart import MIMEMultipart 
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email import Encoders
import os
import ConfigParser


def get_config(env): 
    config = ConfigParser.ConfigParser()
    if env == "DEV":
        config.read(['config/development.cfg']) 
    elif env == "PROD":
        config.read(['config/production.cfg'])
    return config


def mail(to, subject, text, attach=None, config=None): 
    if not config:
        config = get_config("DEV") 
    msg = MIMEMultipart()
    msg['From'] = config.get('email', 'user') 
    msg['To'] = ", ".join(to) 
    msg['Subject'] = subject
    msg.attach(MIMEText(text))
    if attach: 
        part = MIMEBase('application', 'octet-stream')
        part.set_payload(open(attach, 'rb').read()) 
        Encoders.encode_base64(part)
        part.add_header('Content-Disposition',
                        'attachment; filename="%s"' % os.path.basename(attach))
        msg.attach(part)
    mailServer = smtplib.SMTP("smtp.gmail.com", 587) 
    mailServer.ehlo()
    mailServer.starttls()
    mailServer.ehlo()
    mailServer.login(config.get('email', 'user'),
                     config.get('email', 'password'))
    mailServer.sendmail(config.get('email', 'user'), to, msg.as_string())
    mailServer.close()


def example():
    mail(['listof@mydomain.com', 'emails@mydomain.com'],
         "Automate your life: sending emails",
         "Why'd the elephant sit on the marshmallow?",
         attach="my_file.txt") 

Python’s built-in smtplib library gives you a wrapper for SMTP, the standard protocol for sending and receiving email.

Python’s email library helps create email messages and attachments and keeps them in the proper format.

The get_config function loads the configuration from a series of local configuration files. We pass an environment variable, which is expected to be the string "PROD" or "DEV" to signal whether it’s running locally ("DEV") or on our remote production environment ("PROD"). If you only have one environment, you could simply return the only configuration file in your project.

This line uses Python’s ConfigParser to read in the .cfg file and returns config object.

Our mail function takes a list of email addresses as the to variable, the subject and text of the email, an optional attachment, and an optional config argument. The attachment is expected to be the name of a local file. The config should be a Python ConfigParser object.

This line sets the default configuration in case it wasn’t passed. To be safe, we are using the "DEV" configuration.

This code uses the ConfigParser object to pull the email address out of the config file. This keeps the address secure and separate from our repository code.

This code unpacks the list of emails and separates them with commas and a space. It expands the list of email addresses to a string, because that’s what our MIME type expects.

If there is an attachment, this line begins the special handling for MIME multipart standards needed to send attachments.

This code opens and reads the full file using the filename string passed.

If you’re not using Gmail, set these to match your provider’s host and port for SMTP. Those should be easy to identify if there is good documentation. If there isn’t, a simple search for “SMTP settings <your provider name>” should give you the details.

This is some example code to give an idea of what this mail function is expecting. You can see the data types expected (string, list, filename), and the order.

The simple Python built-in libraries smtplib and email help us quickly create and send email messages using their classes and methods. Abstracting some of the other parts of the script (such as saving your email address and password in your config) is an essential part of keeping your script and your repository secure and reusable. A few default settings ensure the script can always send email.

SMS and voice

If you’d like to integrate telephone messages into your alerting, you can use Python to send text messages or make phone calls. Twilio is a very cost-efficient way to do so, with support for messages with media and automated phone calls.

Take a look at how easy it is to send a quick text:

from twilio.rest import TwilioRestClient 
import ConfigParser


def send_text(sender, recipient, text_message, config=None): 
    if not config:
        config = ConfigParser('config/development.cfg')

    client = TwilioRestClient(config.get('twilio', 'account_sid'),
                              config.get('twilio', 'auth_token')) 
    sms = client.sms.messages.create(body=text_message,
                                      to=recipient,
                                      from_=sender) 

def example():
    send_text("+11008675309", "+11088675309", "JENNY!!!!") 

We’ll use the Twilio Python client to interact directly with the Twilio API via Python.

This line defines a function we can use to send a text. We’ll need the sender’s and recipient’s phone numbers (prefaced with country codes) and the simple text message we want to send, and we have the ability to also pass a configuration object. We’ll use the configuration to authorize with the Twilio API.

This code sets up a client object, which will authorize using our Twilio account. When you sign up for Twilio, you’ll receive an account_sid and an auth_token. Put them in the configuration file your script uses, in a section named twilio.

To send a text, this code navigates to the SMS module in our client and calls the message resource’s create method. As documented by Twilio, we can then send a simple text message with only a few parameters.

Twilio works internationally and expects to see international-based dialing numbers. If you’re unsure of the international dialing codes to use, Wikipedia has a good listing.

Chat integration

If you’d like to integrate chat into your alerting, or if your team or collaborators commonly use chat, there are many Python chat toolkits you can use for this purpose. Depending on your chat client and needs, there’s likely a Python or API-based solution, and you can use your knowledge of REST clients to go about connecting and messaging the right people.

If you use HipChat, their API is fairly easy to integrate with your Python application or script. There are several Python libraries to make simple messaging to a chatroom or a person straightforward.

To get started using the HipChat API, you’ll first need to log in and get an API token. You can then use HypChat, a Python library, to send a quick message to a chatroom.

First, install HypChat using pip:

pip install hypchat

Now, send a message using Python!

from hypchat import HypChat
from utils import get_config


def get_client(config):
    client = HypChat(config.get('hipchat', 'token')) 
    return client


def message_room(client, room_name, message):
    try:
        room = client.get_room(room_name) 
        room.message(message) 
    except Exception as e:
        print e 


def main():
    config = get_config('DEV') 
    client = get_client(config)
    message_room(client, 'My Favorite Room', "I'M A ROBOT!")

We use the HypChat library to talk to our chat client. The library initializes a new client using our HipChat token, which we will keep stored in our config files.

This code uses the get_room method, which locates a room matching the string name.

This line sends a message to a room or a user with the message method, and passes it a simple string of what to say.

Always use try...except blocks with API-based libraries in case of connection errors or API changes. This code prints the error, but you’d likely want it logged to fully automate your script.

The get_config function used here is imported from a different script. We follow modular code design by introducing these helper functions and putting them in individual modules for reuse.

If you want to log to chat, you can explore those options with HipLogging. Depending on your needs and how your team works, you can set up your chat logging how you’d like; but it’s nice to know you can always leave a note for someone where they might see it!

If you’d rather use Google Chat, there are some great examples of how to do so using SleekXMPP. You can also use SleekXMPP to send Facebook chat messages.

For Slack messaging, check out the Slack team’s Python client.

For other chat clients, we recommend doing a Google search for “Python <your client name>.” Chances are someone has attempted to connect their Python code with that client, or there’s an API you can use. You know how to use an API from your work in Chapter 13.

With so many options for alerting and messaging about your script’s (and automation’s) success or failure, it’s hard to know which one to use. The important part is to choose a method your or your team regularly use and will see. Prioritizing ease of use and integration with daily life is essential—automation is here to help you save time, not to make you spend more time checking services.

Logging and exceptions

Python-based logging services offer the ability to log to one central service while having your script(s) run on a variety of machines, either local or remote.

One such service with great Python support is Sentry. For a relatively small amount of money per month, you can have access to a dashboard of errors, get alerts sent based on exception thresholds, and monitor the error and exception types you have on a daily, weekly, and monthly basis. The Python client for Sentry is easy to install, configure, and use. If you are using tools like Django, Celery, or even simple Python logging, Sentry has integration points so you don’t need to significantly alter your code to get started. On top of that, the code base is constantly updated and the staff is helpful in case you have questions.

Other options include Airbrake, which originally started as a Ruby-based exception tracker and now supports Python, and Rollbar. It’s a popular market, so there will likely be new ones launched before this book goes to print.

There are also services to pull in and parse your logs, such as Loggly and Logstash. These allow you to monitor your logs on an aggregate level as well as parse, search, and find issues in your logs. They are really only useful if you have enough logs and enough time to review them, but are great for distributed systems with a lot of logging.

Logging and monitoring

If you have distributed machines or you are integrating your script into your company or university’s Python-based server environment, you may want to have robust monitoring of not just Python, but the entire system. There are many services that offer monitoring for system load database traffic, and web applications, as well as automated tasks.

One of the most popular services used for this is New Relic, which can watch your servers and system processes as well as web applications. Using MongoDB and AWS? Or MySQL and Apache? New Relic plug-ins allow you to easily integrate logging for your services into the same dashboards you are using for monitoring server and application health. In addition, they offer a Python agent so you can easily log your Python application (or script) into the same ecosystem. With all of your monitoring in one place, it’s easier to spot issues and set up proper alerting so the right people on your team immediately know about any problems.

Another service for systems and application monitoring is Datadog. Datadog allows you to integrate many services into one dashboard. This saves time and effort and allows you to easily spot errors in your projects, apps, and scripts. The Datadog Python client enables logging of different events you’d like to monitor, but requires a bit of customization.

No matter what monitoring you use, or whether you decide to build your own or use a service, it’s essential to have regular alerting, insight into the services you use, and an understanding of the integrity of your code and automated systems.