Obey the Testing Goat! Do Nothing Until You Have a Test

The Testing Goat is the unofficial mascot of TDD in the Python testing community. It probably means different things to different people, but, to me, the Testing Goat is a voice inside my head that keeps me on the True Path of Testing—like one of those little angels or demons that pop up above your shoulder in the cartoons, but with a very niche set of concerns. I hope, with this book, to install the Testing Goat inside your head too.

We’ve decided to build a website, even if we’re not quite sure what it’s going to do yet. Normally the first step in web development is getting your web framework installed and configured. Download this, install that, configure the other, run the script…but TDD requires a different mindset. When you’re doing TDD, you always have the Testing Goat inside you—single-minded as goats are—bleating “Test first, test first!”

In TDD the first step is always the same: write a test.

First we write the test; then we run it and check that it fails as expected. Only then do we go ahead and build some of our app. Repeat that to yourself in a goat-like voice. I know I do.

Another thing about goats is that they take one step at a time. That’s why they seldom fall off mountains, see, no matter how steep they are. As you can see in Figure 1-1.

Figure 1-1. Goats are more agile than you think (source: Caitlin Stewart, on Flickr)

We’ll proceed with nice small steps; we’re going to use Django, which is a popular Python web framework, to build our app.

The first thing we want to do is check that we’ve got Django installed, and that it’s ready for us to work with. The way we’ll check is by confirming that we can spin up Django’s development server and actually see it serving up a web page, in our web browser, on our local PC. We’ll use the Selenium browser automation tool for this.

Create a new Python file called functional_tests.py, wherever you want to keep the code for your project, and enter the following code. If you feel like making a few little goat noises as you do it, it may help:

from selenium import webdriver

browser = webdriver.Firefox()
browser.get('http://localhost:8000')

assert 'Django' in browser.title

That’s our first functional test (FT); I’ll talk more about what I mean by functional tests, and how they contrast with unit tests, in a bit. For now, it’s enough to assure ourselves that we understand what it’s doing:

• Starting a Selenium “webdriver” to pop up a real Firefox browser window

• Using it to open up a web page which we’re expecting to be served from the local PC

• Checking (making a test assertion) that the page has the word “Django” in its title

Let’s try running it:

$ python functional_tests.py
  File ".../selenium/webdriver/remote/webdriver.py", line 324, in get
    self.execute(Command.GET, {'url': url})
  File ".../selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File ".../selenium/webdriver/remote/errorhandler.py", line 242, in
check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Reached error page: abo
ut:neterror?e=connectionFailure&u=http%3A//localhost%3A8000/[...]

You should see a browser window pop up and try to open localhost:8000, and show the “Unable to connect” error page. If you switch back to your console, you’ll see the big ugly error message, telling us that Selenium hit an error page. And then, you will probably be irritated at the fact that it left the Firefox window lying around your desktop for you to tidy up. We’ll fix that later!

For now though, we have a failing test, so that means we’re allowed to start building our app.

Getting Django Up and Running

Since you’ve definitely read “Prerequisites and Assumptions” by now, you’ve already got Django installed. The first step in getting Django up and running is to create a project, which will be the main container for our site. Django provides a little command-line tool for this:

$ django-admin.py startproject superlists .

Don’t forget that “.” at the end; it’s important!

That will create a file called manage.py in your current folder, and a subfolder called superlists, with more stuff inside it:

├── functional_tests.py
├── geckodriver.log
├── manage.py
├── superlists
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── virtualenv
    ├── [...]

The superlists folder is intended for stuff that applies to the whole project—like settings.py, for example, which is used to store global configuration information for the site.

But the main thing to notice is manage.py. That’s Django’s Swiss Army knife, and one of the things it can do is run a development server. Let’s try that now:

$ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have 13 unapplied migration(s). Your project may not work properly until
you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

Django version 1.11.3, using settings 'superlists.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

That’s Django’s development server now up and running on our machine.

Leave it there and open another command shell. Navigate to your project folder, activate your virtualenv, and then try running our test again:

$ python functional_tests.py
$

Not much action on the command line, but you should notice two things: firstly, there was no ugly AssertionError and secondly, the Firefox window that Selenium popped up had a different-looking page on it.

Well, it may not look like much, but that was our first ever passing test! Hooray!

If it all feels a bit too much like magic, like it wasn’t quite real, why not go and take a look at the dev server manually, by opening a web browser yourself and visiting http://localhost:8000? You should see something like Figure 1-2.

You can quit the development server now if you like, back in the original shell, using Ctrl-C.

Figure 1-2. It worked!

Starting a Git Repository

There’s one last thing to do before we finish the chapter: start to commit our work to a version control system (VCS). If you’re an experienced programmer you don’t need to hear me preaching about version control, but if you’re new to it please believe me when I say that VCS is a must-have. As soon as your project gets to be more than a few weeks old and a few lines of code, having a tool available to look back over old versions of code, revert changes, explore new ideas safely, even just as a backup…boy. TDD goes hand in hand with version control, so I want to make sure I impart how it fits into the workflow.

So, our first commit! If anything it’s a bit late; shame on us. We’re using Git as our VCS, ’cos it’s the best.

Let’s start by doing the git init to start the repository:

$ ls
db.sqlite3
functional_tests.py
geckodriver.log
manage.py
superlists
virtualenv

$ git init .
Initialised empty Git repository in ...python-tdd-book/.git/

Now let’s take a look and see what files we want to commit:

$ ls
db.sqlite3
functional_tests.py
geckodriver.log
manage.py
superlists
virtualenv

There are a few things in here that we don’t want under version control: db.sqlite3 is the database file, geckodriver.log contains Selenium debug output, and finally our virtualenv shouldn’t be in git either. We’ll add all of them to a special file called .gitignore which, um, tells Git what to ignore:

$ echo "db.sqlite3" >> .gitignore
$ echo "geckodriver.log" >> .gitignore
$ echo "virtualenv" >> .gitignore

Next we can add the rest of the contents of the current folder, “.”:

$ git add .
$ git status
On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

        new file:   .gitignore
        new file:   functional_tests.py
        new file:   manage.py
        new file:   superlists/__init__.py
        new file:   superlists/__pycache__/__init__.cpython-36.pyc
        new file:   superlists/__pycache__/settings.cpython-36.pyc
        new file:   superlists/__pycache__/urls.cpython-36.pyc
        new file:   superlists/__pycache__/wsgi.cpython-36.pyc
        new file:   superlists/settings.py
        new file:   superlists/urls.py
        new file:   superlists/wsgi.py

Oops! We’ve got a bunch of .pyc files in there; it’s pointless to commit those. Let’s remove them from Git and add them to .gitignore too:

$ git rm -r --cached superlists/__pycache__
rm 'superlists/__pycache__/__init__.cpython-36.pyc'
rm 'superlists/__pycache__/settings.cpython-36.pyc'
rm 'superlists/__pycache__/urls.cpython-36.pyc'
rm 'superlists/__pycache__/wsgi.cpython-36.pyc'
$ echo "__pycache__" >> .gitignore
$ echo "*.pyc" >> .gitignore

Now let’s see where we are… (You’ll see I’m using git status a lot—so much so that I often alias it to git st…I’m not telling you how to do that though; I leave you to discover the secrets of Git aliases on your own!):

$ git status
On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

        new file:   .gitignore
        new file:   functional_tests.py
        new file:   manage.py
        new file:   superlists/__init__.py
        new file:   superlists/settings.py
        new file:   superlists/urls.py
        new file:   superlists/wsgi.py

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   .gitignore

Looking good—we’re ready to do our first commit!

$ git add .gitignore
$ git commit

When you type git commit, it will pop up an editor window for you to write your commit message in. Mine looked like Figure 1-3.¹

Figure 1-3. First Git commit

That’s it for the VCS lecture. Congratulations! You’ve written a functional test using Selenium, and you’ve gotten Django installed and running, in a certifiable, test-first, goat-approved TDD way. Give yourself a well-deserved pat on the back before moving on to Chapter 2.

The Python Standard Library’s unittest Module

There are a couple of little annoyances we should probably deal with. Firstly, the message “AssertionError” isn’t very helpful—it would be nice if the test told us what it actually found as the browser title. Also, it’s left a Firefox window hanging around the desktop, so it would be nice if that got cleared up for us automatically.

One option would be to use the second parameter to the assert keyword, something like:

assert 'To-Do' in browser.title, "Browser title was " + browser.title

And we could also use a try/finally to clean up the old Firefox window. But these sorts of problems are quite common in testing, and there are some ready-made solutions for us in the standard library’s unittest module. Let’s use that! In functional_tests.py:

from selenium import webdriver
import unittest

class NewVisitorTest(unittest.TestCase):  

    def setUp(self):  
        self.browser = webdriver.Firefox()

    def tearDown(self):  
        self.browser.quit()

    def test_can_start_a_list_and_retrieve_it_later(self):  
        # Edith has heard about a cool new online to-do app. She goes
        # to check out its homepage
        self.browser.get('http://localhost:8000')

        # She notices the page title and header mention to-do lists
        self.assertIn('To-Do', self.browser.title)  
        self.fail('Finish the test!')  

        # She is invited to enter a to-do item straight away
        [...rest of comments as before]

if __name__ == '__main__':  
    unittest.main(warnings='ignore')  

You’ll probably notice a few things here:

Tests are organised into classes, which inherit from unittest.TestCase.

The main body of the test is in a method called test_can_start_​a_list_and_retrieve_it_later. Any method whose name starts with test is a test method, and will be run by the test runner. You can have more than one test_ method per class. Nice descriptive names for our test methods are a good idea too.

setUp and tearDown are special methods which get run before and after each test. I’m using them to start and stop our browser—note that they’re a bit like a try/except, in that tearDown will run even if there’s an error during the test itself.¹ No more Firefox windows left lying around!

We use self.assertIn instead of just assert to make our test assertions. unittest provides lots of helper functions like this to make test assertions, like assertEqual, assertTrue, assertFalse, and so on. You can find more in the unittest documentation.

self.fail just fails no matter what, producing the error message given. I’m using it as a reminder to finish the test.

Finally, we have the if __name__ == '__main__' clause (if you’ve not seen it before, that’s how a Python script checks if it’s been executed from the command line, rather than just imported by another script). We call unittest.main(), which launches the unittest test runner, which will automatically find test classes and methods in the file and run them.

warnings='ignore' suppresses a superfluous ResourceWarning which was being emitted at the time of writing. It may have disappeared by the time you read this; feel free to try removing it!

Let’s try it!

$ python functional_tests.py
F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later (__main__.NewVisitorTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "functional_tests.py", line 18, in
test_can_start_a_list_and_retrieve_it_later
    self.assertIn('To-Do', self.browser.title)
AssertionError: 'To-Do' not found in 'Welcome to Django'

 ---------------------------------------------------------------------
Ran 1 test in 1.747s

FAILED (failures=1)

That’s a bit nicer, isn’t it? It tidied up our Firefox window, it gives us a nicely formatted report of how many tests were run and how many failed, and the assertIn has given us a helpful error message with useful debugging info. Bonzer!

Commit

This is a good point to do a commit; it’s a nicely self-contained change. We’ve expanded our functional test to include comments that describe the task we’re setting ourselves, our minimum viable to-do list. We’ve also rewritten it to use the Python unittest module and its various testing helper functions.

Do a git status—that should assure you that the only file that has changed is functional_tests.py. Then do a git diff, which shows you the difference between the last commit and what’s currently on disk. That should tell you that functional_tests.py has changed quite substantially:

$ git diff
diff --git a/functional_tests.py b/functional_tests.py
index d333591..b0f22dc 100644
--- a/functional_tests.py
+++ b/functional_tests.py
@@ -1,6 +1,45 @@
 from selenium import webdriver
+import unittest

-browser = webdriver.Firefox()
-browser.get('http://localhost:8000')
+class NewVisitorTest(unittest.TestCase):

-assert 'Django' in browser.title
+    def setUp(self):
+        self.browser = webdriver.Firefox()
+
+    def tearDown(self):
+        self.browser.quit()
[...]

Now let’s do a:

$ git commit -a

The -a means “automatically add any changes to tracked files” (i.e., any files that we’ve committed before). It won’t add any brand new files (you have to explicitly git add them yourself), but often, as in this case, there aren’t any new files, so it’s a useful shortcut.

When the editor pops up, add a descriptive commit message, like “First FT specced out in comments, and now uses unittest.”

Now we’re in an excellent position to start writing some real code for our lists app. Read on!

Our First Django App, and Our First Unit Test

Django encourages you to structure your code into apps: the theory is that one project can have many apps, you can use third-party apps developed by other people, and you might even reuse one of your own apps in a different project…although I admit I’ve never actually managed it myself! Still, apps are a good way to keep your code organised.

Let’s start an app for our to-do lists:

$ python manage.py startapp lists

That will create a folder called lists, next to manage.py and the existing superlists folder , and within it a number of placeholder files for things like models, views, and, of immediate interest to us, tests:

.
├── db.sqlite3
├── functional_tests.py
├── lists
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── manage.py
├── superlists
│   ├── __init__.py
│   ├── __pycache__
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── virtualenv
    ├── [...]

Unit Tests, and How They Differ from Functional Tests

As with so many of the labels we put on things, the line between unit tests and functional tests can become a little blurry at times. The basic distinction, though, is that functional tests test the application from the outside, from the point of view of the user. Unit tests test the application from the inside, from the point of view of the programmer.

The TDD approach I’m following wants our application to be covered by both types of test. Our workflow will look a bit like this:

1. We start by writing a functional test, describing the new functionality from the user’s point of view.

2. Once we have a functional test that fails, we start to think about how to write code that can get it to pass (or at least to get past its current failure). We now use one or more unit tests to define how we want our code to behave—the idea is that each line of production code we write should be tested by (at least) one of our unit tests.

3. Once we have a failing unit test, we write the smallest amount of application code we can, just enough to get the unit test to pass. We may iterate between steps 2 and 3 a few times, until we think the functional test will get a little further.

4. Now we can rerun our functional tests and see if they pass, or get a little further. That may prompt us to write some new unit tests, and some new code, and so on.

You can see that, all the way through, the functional tests are driving what development we do from a high level, while the unit tests drive what we do at a low level.

Does that seem slightly redundant? Sometimes it can feel that way, but functional tests and unit tests do really have very different objectives, and they will usually end up looking quite different.

Enough theory for now—let’s see how it looks in practice.

Unit Testing in Django

Let’s see how to write a unit test for our home page view. Open up the new file at lists/tests.py, and you’ll see something like this:

from django.test import TestCase

# Create your tests here.

Django has helpfully suggested we use a special version of TestCase, which it provides. It’s an augmented version of the standard unittest.TestCase, with some additional Django-specific features, which we’ll discover over the next few chapters.

You’ve already seen that the TDD cycle involves starting with a test that fails, then writing code to get it to pass. Well, before we can even get that far, we want to know that the unit test we’re writing will definitely be run by our automated test runner, whatever it is. In the case of functional_tests.py, we’re running it directly, but this file made by Django is a bit more like magic. So, just to make sure, let’s make a deliberately silly failing test:

from django.test import TestCase

class SmokeTest(TestCase):

    def test_bad_maths(self):
        self.assertEqual(1 + 1, 3)

Now let’s invoke this mysterious Django test runner. As usual, it’s a manage.py command:

$ python manage.py test
Creating test database for alias 'default'...
F
======================================================================
FAIL: test_bad_maths (lists.tests.SmokeTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "...python-tdd-book/lists/tests.py", line 6, in test_bad_maths
    self.assertEqual(1 + 1, 3)
AssertionError: 2 != 3

 ---------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...

Excellent. The machinery seems to be working. This is a good point for a commit:

$ git status  # should show you lists/ is untracked
$ git add lists
$ git diff --staged  # will show you the diff that you're about to commit
$ git commit -m "Add app for lists, with deliberately failing unit test"

As you’ve no doubt guessed, the -m flag lets you pass in a commit message at the command line, so you don’t need to use an editor. It’s up to you to pick the way you like to use the Git command line; I’ll just show you the main ones I’ve seen used. The key rule is: make sure you always review what you’re about to commit before you do it.

Django’s MVC, URLs, and View Functions

Django is structured along a classic Model-View-Controller (MVC) pattern. Well, broadly. It definitely does have models, but its views are more like a controller, and it’s the templates that are actually the view part, but the general idea is there. If you’re interested, you can look up the finer points of the discussion in the Django FAQs.

Irrespective of any of that, as with any web server, Django’s main job is to decide what to do when a user asks for a particular URL on our site. Django’s workflow goes something like this:

1. An HTTP request comes in for a particular URL.

2. Django uses some rules to decide which view function should deal with the request (this is referred to as resolving the URL).

3. The view function processes the request and returns an HTTP response.

So we want to test two things:

• Can we resolve the URL for the root of the site (“/”) to a particular view function we’ve made?

• Can we make this view function return some HTML which will get the functional test to pass?

Let’s start with the first. Open up lists/tests.py, and change our silly test to something like this:

from django.urls import resolve
from django.test import TestCase
from lists.views import home_page  

class HomePageTest(TestCase):

    def test_root_url_resolves_to_home_page_view(self):
        found = resolve('/')  
        self.assertEqual(found.func, home_page)  

What’s going on here?

resolve is the function Django uses internally to resolve URLs and find what view function they should map to. We’re checking that resolve, when called with “/”, the root of the site, finds a function called home_page.

What function is that? It’s the view function we’re going to write next, which will actually return the HTML we want. You can see from the import that we’re planning to store it in lists/views.py.

So, what do you think will happen when we run the tests?

$ python manage.py test
ImportError: cannot import name 'home_page'

It’s a very predictable and uninteresting error: we tried to import something we haven’t even written yet. But it’s still good news—for the purposes of TDD, an exception which was predicted counts as an expected failure. Since we have both a failing functional test and a failing unit test, we have the Testing Goat’s full blessing to code away.

At Last! We Actually Write Some Application Code!

It is exciting, isn’t it? Be warned, TDD means that long periods of anticipation are only defused very gradually, and by tiny increments. Especially since we’re learning and only just starting out, we only allow ourselves to change (or add) one line of code at a time—and each time, we make just the minimal change required to address the current test failure.

I’m being deliberately extreme here, but what’s our current test failure? We can’t import home_page from lists.views? OK, let’s fix that—and only that. In lists/views.py:

from django.shortcuts import render

# Create your views here.
home_page = None

“You must be joking!” I can hear you say.

I can hear you because it’s what I used to say (with feeling) when my colleagues first demonstrated TDD to me. Well, bear with me, and we’ll talk about whether or not this is all taking it too far in a little while. But for now, let yourself follow along, even if it’s with some exasperation, and see if our tests can help us write the correct code, one tiny step at a time.

We run the tests again:

$ python manage.py test
Creating test database for alias 'default'...
E
======================================================================
ERROR: test_root_url_resolves_to_home_page_view (lists.tests.HomePageTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "...python-tdd-book/lists/tests.py", line 8, in
test_root_url_resolves_to_home_page_view
    found = resolve('/')
  File ".../django/urls/base.py", line 27, in resolve
    return get_resolver(urlconf).resolve(path)
  File ".../django/urls/resolvers.py", line 392, in resolve
    raise Resolver404({'tried': tried, 'path': new_path})
django.urls.exceptions.Resolver404: {'tried': [[<RegexURLResolver
<RegexURLPattern list> (admin:admin) ^admin/>]], 'path': ''}

 ---------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...

urls.py

Our tests are telling us that we need a URL mapping. Django uses a file called urls.py to map URLs to view functions. There’s a main urls.py for the whole site in the superlists/superlists folder. Let’s go take a look:

"""superlists URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/1.11/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  url(r'^$', views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  url(r'^$', Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.conf.urls import url, include
    2. Add a URL to urlpatterns:  url(r'^blog/', include('blog.urls'))
"""
from django.conf.urls import url
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls),
]

As usual, lots of helpful comments and default suggestions from Django.

A url entry starts with a regular expression that defines which URLs it applies to, and goes on to say where it should send those requests—either to a view function you’ve imported, or maybe to another urls.py file somewhere else.

The first example entry has the regular expression ^$, which means an empty string—could this be the same as the root of our site, which we’ve been testing with “/”? Let’s find out—what happens if we include it?

We’ll also get rid of the admin URL, because we won’t be using the Django admin site for now:

from django.conf.urls import url
from lists import views

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
]

Run the unit tests again, with python manage.py test:

[...]
TypeError: view must be a callable or a list/tuple in the case of include().

That’s progress! We’re no longer getting a 404.

The traceback is messy, but the message at the end is telling us what’s going on: the unit tests have actually made the link between the URL “/” and the home_page = None in lists/views.py, and are now complaining that the home_page view is not callable. And that gives us a justification for changing it from being None to being an actual function. Every single code change is driven by the tests!

Back in lists/views.py:

from django.shortcuts import render

# Create your views here.
def home_page():
    pass

And now?

$ python manage.py test
Creating test database for alias 'default'...
.
 ---------------------------------------------------------------------
Ran 1 test in 0.003s

OK
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...

Hooray! Our first ever unit test pass! That’s so momentous that I think it’s worthy of a commit:

$ git diff  # should show changes to urls.py, tests.py, and views.py
$ git commit -am "First unit test and url mapping, dummy view"

That was the last variation on git commit I’ll show, the a and m flags together, which adds all changes to tracked files and uses the commit message from the command line.

Unit Testing a View

On to writing a test for our view, so that it can be something more than a do-nothing function, and instead be a function that returns a real response with HTML to the browser. Open up lists/tests.py, and add a new test method. I’ll explain each bit:

from django.urls import resolve
from django.test import TestCase
from django.http import HttpRequest

from lists.views import home_page


class HomePageTest(TestCase):

    def test_root_url_resolves_to_home_page_view(self):
        found = resolve('/')
        self.assertEqual(found.func, home_page)


    def test_home_page_returns_correct_html(self):
        request = HttpRequest()  
        response = home_page(request)  
        html = response.content.decode('utf8')  
        self.assertTrue(html.startswith('<html>'))  
        self.assertIn('<title>To-Do lists</title>', html)  
        self.assertTrue(html.endswith('</html>'))  

What’s going on in this new test?

We create an HttpRequest object, which is what Django will see when a user’s browser asks for a page.

We pass it to our home_page view, which gives us a response. You won’t be surprised to hear that this object is an instance of a class called HttpResponse.

Then, we extract the .content of the response. These are the raw bytes, the ones and zeros that would be sent down the wire to the user’s browser. We call .decode() to convert them into the string of HTML that’s being sent to the user.

We want it to start with an <html> tag which gets closed at the end.

And we want a <title> tag somewhere in the middle, with the words “To-Do lists” in it—because that’s what we specified in our functional test.

Once again, the unit test is driven by the functional test, but it’s also much closer to the actual code—we’re thinking like programmers now.

Let’s run the unit tests now and see how we get on:

TypeError: home_page() takes 0 positional arguments but 1 was given

$ python functional_tests.py
F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later (__main__.NewVisitorTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "functional_tests.py", line 19, in
test_can_start_a_list_and_retrieve_it_later
    self.fail('Finish the test!')
AssertionError: Finish the test!

 ---------------------------------------------------------------------
Ran 1 test in 1.609s

FAILED (failures=1)

$ git diff  # should show our new test in tests.py, and the view in views.py
$ git commit -am "Basic view now returns minimal HTML"

$ git log --oneline
a6e6cc9 Basic view now returns minimal HTML
450c0f3 First unit test and url mapping, dummy view
ea2b037 Add app for lists, with deliberately failing unit test
[...]

Programming Is Like Pulling a Bucket of Water Up from a Well

Ultimately, programming is hard. Often, we are smart, so we succeed. TDD is there to help us out when we’re not so smart. Kent Beck (who basically invented TDD) uses the metaphor of lifting a bucket of water out of a well with a rope: when the well isn’t too deep, and the bucket isn’t very full, it’s easy. And even lifting a full bucket is pretty easy at first. But after a while, you’re going to get tired. TDD is like having a ratchet that lets you save your progress, take a break, and make sure you never slip backwards. That way you don’t have to be smart all the time.

Figure 4-1. Test ALL the things (original illustration source: Allie Brosh, Hyperbole and a Half)

OK, perhaps in general, you’re prepared to concede that TDD is a good idea, but maybe you still think I’m overdoing it? Testing the tiniest thing, and taking ridiculously many small steps?

TDD is a discipline, and that means it’s not something that comes naturally; because many of the payoffs aren’t immediate but only come in the longer term, you have to force yourself to do it in the moment. That’s what the image of the Testing Goat is supposed to illustrate—you need to be a bit bloody-minded about it.

Now, back to our onions.

The “Don’t Test Constants” Rule, and Templates to the Rescue

Let’s take a look at our unit tests, lists/tests.py. Currently we’re looking for specific HTML strings, but that’s not a particularly efficient way of testing HTML. In general, one of the rules of unit testing is Don’t test constants, and testing HTML as text is a lot like testing a constant.

In other words, if you have some code that says:

wibble = 3

There’s not much point in a test that says:

from myprogram import wibble
assert wibble == 3

Unit tests are really about testing logic, flow control, and configuration. Making assertions about exactly what sequence of characters we have in our HTML strings isn’t doing that.

What’s more, mangling raw strings in Python really isn’t a great way of dealing with HTML. There’s a much better solution, which is to use templates. Quite apart from anything else, if we can keep HTML to one side in a file whose name ends in .html, we’ll get better syntax highlighting! There are lots of Python templating frameworks out there, and Django has its own which works very well. Let’s use that.

$ python manage.py test
[...]
OK

<html>
    <title>To-Do lists</title>
</html>

from django.shortcuts import render

def home_page(request):
    return render(request, 'home.html')

$ python manage.py test
[...]
======================================================================
ERROR: test_home_page_returns_correct_html (lists.tests.HomePageTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "...python-tdd-book/lists/tests.py", line 17, in
test_home_page_returns_correct_html
    response = home_page(request)
  File "...python-tdd-book/lists/views.py", line 5, in home_page
    return render(request, 'home.html')
  File "/usr/local/lib/python3.6/dist-packages/django/shortcuts.py", line 48,
in render
    return HttpResponse(loader.render_to_string(*args, **kwargs),
  File "/usr/local/lib/python3.6/dist-packages/django/template/loader.py", line
170, in render_to_string
    t = get_template(template_name, dirs)
  File "/usr/local/lib/python3.6/dist-packages/django/template/loader.py", line
144, in get_template
    template, origin = find_template(template_name, dirs)
  File "/usr/local/lib/python3.6/dist-packages/django/template/loader.py", line
136, in find_template
    raise TemplateDoesNotExist(name)
django.template.base.TemplateDoesNotExist: home.html

 ---------------------------------------------------------------------
Ran 2 tests in 0.004s

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'lists',
]

$ python manage.py test
    [...]
    self.assertTrue(html.endswith('</html>'))
AssertionError: False is not true

self.assertTrue(html.strip().endswith('</html>'))

$ python manage.py test
[...]
OK

from django.template.loader import render_to_string
[...]

    def test_home_page_returns_correct_html(self):
        request = HttpRequest()
        response = home_page(request)
        html = response.content.decode('utf8')
        expected_html = render_to_string('home.html')
        self.assertEqual(html, expected_html)

    def test_home_page_returns_correct_html(self):
        response = self.client.get('/')  

        html = response.content.decode('utf8')  
        self.assertTrue(html.startswith('<html>'))
        self.assertIn('<title>To-Do lists</title>', html)
        self.assertTrue(html.strip().endswith('</html>'))

        self.assertTemplateUsed(response, 'home.html')  

Ran 2 tests in 0.016s

OK

        self.assertTemplateUsed(response, 'wrong.html')

AssertionError: False is not true : Template 'wrong.html' was not a template
used to render the response. Actual template(s) used: home.html

from django.test import TestCase

class HomePageTest(TestCase):

    def test_uses_home_template(self):
        response = self.client.get('/')
        self.assertTemplateUsed(response, 'home.html')

On Refactoring

That was an absolutely trivial example of refactoring. But, as Kent Beck puts it in Test-Driven Development: By Example, “Am I recommending that you actually work this way? No. I’m recommending that you be able to work this way”.

In fact, as I was writing this my first instinct was to dive in and change the test first—make it use the assertTemplateUsed function straight away; delete the three superfluous assertions, leaving just a check of the contents against the expected render; and then go ahead and make the code change. But notice how that actually would have left space for me to break things: I could have defined the template as containing any arbitrary string, instead of the string with the right <html> and <title> tags.

There’s always a tendency to skip ahead a couple of steps, to make a couple of tweaks to the behaviour while you’re refactoring, but pretty soon you’ve got changes to half a dozen different files, you’ve totally lost track of where you are, and nothing works any more. If you don’t want to end up like Refactoring Cat (Figure 4-2), stick to small steps; keep refactoring and functionality changes entirely separate.

Figure 4-2. Refactoring Cat—be sure to look up the full animated GIF (source: 4GIFs.com)

It’s a good idea to do a commit after any refactoring:

$ git status # see tests.py, views.py, settings.py, + new templates folder
$ git add .  # will also add the untracked templates folder
$ git diff --staged # review the changes we're about to commit
$ git commit -m "Refactor home page view to use a template"

A Little More of Our Front Page

In the meantime, our functional test is still failing. Let’s now make an actual code change to get it passing. Because our HTML is now in a template, we can feel free to make changes to it, without needing to write any extra unit tests. We wanted an <h1>:

<html>
    <head>
        <title>To-Do lists</title>
    </head>
    <body>
        <h1>Your To-Do list</h1>
    </body>
</html>

Let’s see if our functional test likes it a little better:

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_new_item"]

OK…

    [...]
        <h1>Your To-Do list</h1>
        <input id="id_new_item" />
    </body>
    [...]

And now?

AssertionError: '' != 'Enter a to-do item'

We add our placeholder text…

    <input id="id_new_item" placeholder="Enter a to-do item" />

Which gives:

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]

So we can go ahead and put the table onto the page. At this stage it’ll just be empty…

    <input id="id_new_item" placeholder="Enter a to-do item" />
    <table id="id_list_table">
    </table>
</body>

Now what does the FT say?

  File "functional_tests.py", line 43, in
test_can_start_a_list_and_retrieve_it_later
    any(row.text == '1: Buy peacock feathers' for row in rows)
AssertionError: False is not true

Slightly cryptic. We can use the line number to track it down, and it turns out it’s that any function I was so smug about earlier—or, more precisely, the assertTrue, which doesn’t have a very explicit failure message. We can pass a custom error message as an argument to most assertX methods in unittest:

    self.assertTrue(
        any(row.text == '1: Buy peacock feathers' for row in rows),
        "New to-do item did not appear in table"
    )

If you run the FT again, you should see our message:

AssertionError: False is not true : New to-do item did not appear in table

But now, to get this to pass, we will need to actually process the user’s form submission. And that’s a topic for the next chapter.

For now let’s do a commit:

$ git diff
$ git commit -am "Front page HTML now generated from a template"

Thanks to a bit of refactoring, we’ve got our view set up to render a template, we’ve stopped testing constants, and we’re now well placed to start processing user input.

Recap: The TDD Process

We’ve now seen all the main aspects of the TDD process, in practice:

• Functional tests

• Unit tests

• The unit-test/code cycle

• Refactoring

It’s time for a little recap, and perhaps even some flowcharts. Forgive me, years misspent as a management consultant have ruined me. On the plus side, it will feature recursion.

What is the overall TDD process? See Figure 4-3.

We write a test. We run the test and see it fail. We write some minimal code to get it a little further. We rerun the test and repeat until it passes. Then, optionally, we might refactor our code, using our tests to make sure we don’t break anything.

Figure 4-3. Overall TDD process

But how does this apply when we have functional tests and unit tests? Well, you can think of the functional test as being a high-level view of the cycle, where “writing the code” to get the functional tests to pass actually involves using another, smaller TDD cycle which uses unit tests. See Figure 4-4.

We write a functional test and see it fail. Then, the process of “writing code” to get it to pass is a mini-TDD cycle of its own: we write one or more unit tests, and go into the unit-test/code cycle until the unit tests pass. Then, we go back to our FT to check that it gets a little further, and we can write a bit more of our application—using more unit tests, and so on.

What about refactoring, in the context of functional tests? Well, that means we use the functional test to check that we’ve preserved the behaviour of our application, but we can change or add and remove unit tests, and use a unit test cycle to actually change the implementation.

The functional tests are the ultimate judge of whether your application works or not. The unit tests are a tool to help you along the way.

This way of looking at things is sometimes called “Double-Loop TDD”. One of my eminent tech reviewers, Emily Bache, wrote a blog post on the topic, which I recommend for a different perspective.

Figure 4-4. The TDD process with functional and unit tests

We’ll explore all of the different parts of this workflow in more detail over the coming chapters.

Wiring Up Our Form to Send a POST Request

At the end of the last chapter, the tests were telling us we weren’t able to save the user’s input. For now, we’ll use a standard HTML POST request. A little boring, but also nice and easy to deliver—we can use all sorts of sexy HTML5 and JavaScript later in the book.

To get our browser to send a POST request, we need to do two things:

1. Give the <input> element a name= attribute.

2. Wrap it in a <form> tag with method="POST".

Let’s adjust our template at lists/templates/home.html:

<h1>Your To-Do list</h1>
<form method="POST">
    <input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
</form>

<table id="id_list_table">

Now, running our FTs gives us a slightly cryptic, unexpected error:

$ python functional_tests.py
[...]
Traceback (most recent call last):
  File "functional_tests.py", line 40, in
test_can_start_a_list_and_retrieve_it_later
    table = self.browser.find_element_by_id('id_list_table')
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]

When a functional test fails with an unexpected failure, there are several things we can do to debug it:

• Add print statements, to show, for example, what the current page text is.

• Improve the error message to show more info about the current state.

• Manually visit the site yourself.

• Use time.sleep to pause the test during execution.¹

We’ll look at all of these over the course of this book, but the time.sleep option is one I find myself using very often. Let’s try it now.

Conveniently, we’ve already got a sleep just before the error occurs; let’s just extend it a little:

    # When she hits enter, the page updates, and now the page lists
    # "1: Buy peacock feathers" as an item in a to-do list table
    inputbox.send_keys(Keys.ENTER)
    time.sleep(10)

    table = self.browser.find_element_by_id('id_list_table')

Depending on how fast Selenium runs on your PC, you may have caught a glimpse of this already, but when we run the functional tests again, we’ve got time to see what’s going on: you should see a page that looks like Figure 5-1, with lots of Django debug information.

Figure 5-1. Django DEBUG page showing CSRF error

Django’s CSRF protection involves placing a little auto-generated token into each generated form, to be able to identify POST requests as having come from the original site. So far our template has been pure HTML, and in this step we make the first use of Django’s template magic. To add the CSRF token we use a template tag, which has the curly-bracket/percent syntax, {% ... %}—famous for being the world’s most annoying two-key touch-typing combination:

<form method="POST">
    <input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
    {% csrf_token %}
</form>

Django will substitute that during rendering with an <input type="hidden"> containing the CSRF token. Rerunning the functional test will now give us an expected failure:

AssertionError: False is not true : New to-do item did not appear in table

Since our long time.sleep is still there, the test will pause on the final screen, showing us that the new item text disappears after the form is submitted, and the page refreshes to show an empty form again. That’s because we haven’t wired up our server to deal with the POST request yet—it just ignores it and displays the normal home page.

We can put our normal short time.sleep back now though:

    # "1: Buy peacock feathers" as an item in a to-do list table
    inputbox.send_keys(Keys.ENTER)
    time.sleep(1)

    table = self.browser.find_element_by_id('id_list_table')

Processing a POST Request on the Server

Because we haven’t specified an action= attribute in the form, it is submitting back to the same URL it was rendered from by default (i.e., /), which is dealt with by our home_page function. Let’s adapt the view to be able to deal with a POST request.

That means a new unit test for the home_page view. Open up lists/tests.py, and add a new method to HomePageTest:

def test_uses_home_template(self):
    response = self.client.get('/')
    self.assertTemplateUsed(response, 'home.html')


def test_can_save_a_POST_request(self):
    response = self.client.post('/', data={'item_text': 'A new list item'})
    self.assertIn('A new list item', response.content.decode())

To do a POST, we call self.client.post, and as you can see it takes a data argument which contains the form data we want to send. Then we check that the text from our POST request ends up in the rendered HTML. That gives us our expected fail:

$ python manage.py test
[...]
AssertionError: 'A new list item' not found in '<html>\n    <head>\n
<title>To-Do lists</title>\n    </head>\n    <body>\n        <h1>Your To-Do
list</h1>\n        <form method="POST">\n            <input name="item_text"
[...]
</body>\n</html>\n'

We can get the test to pass by adding an if and providing a different code path for POST requests. In typical TDD style, we start with a deliberately silly return value:

from django.http import HttpResponse
from django.shortcuts import render

def home_page(request):
    if request.method == 'POST':
        return HttpResponse(request.POST['item_text'])
    return render(request, 'home.html')

That gets our unit tests passing, but it’s not really what we want. What we really want to do is add the POST submission to the table in the home page template.

Passing Python Variables to Be Rendered in the Template

We’ve already had a hint of it, and now it’s time to start to get to know the real power of the Django template syntax, which is to pass variables from our Python view code into HTML templates.

Let’s start by seeing how the template syntax lets us include a Python object in our template. The notation is {{ ... }}, which displays the object as a string:

<body>
    <h1>Your To-Do list</h1>
    <form method="POST">
        <input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
        {% csrf_token %}
    </form>

    <table id="id_list_table">
        <tr><td>{{ new_item_text }}</td></tr>  
    </table>
</body>

Here’s our template variable. new_item_text will be the variable name for the user input we display in the template, to help distinguish it from item_text, which is the name of the form field which we use in the POST request. That just reminds us that transforming the one into the other doesn’t happen automatically; it’s something we do ourselves in the view…

Let’s adjust our unit test so that it checks whether we are still using the template:

    def test_can_save_a_POST_request(self):
        response = self.client.post('/', data={'item_text': 'A new list item'})
        self.assertIn('A new list item', response.content.decode())
        self.assertTemplateUsed(response, 'home.html')

And that will fail as expected:

AssertionError: No templates used to render the response

Good, our deliberately silly return value is now no longer fooling our tests, so we are allowed to rewrite our view, and tell it to pass the POST parameter to the template. The render function takes, as its third argument, a dictionary which maps template variable names to their values, so we can use it for the POST case as well as the normal case. Let’s simplify our view right down to:

def home_page(request):
    return render(request, 'home.html', {
        'new_item_text': request.POST['item_text'],
    })

Running the unit tests again:

ERROR: test_uses_home_template (lists.tests.HomePageTest)
[...]
  File "...python-tdd-book/lists/views.py", line 5, in home_page
    'new_item_text': request.POST['item_text'],
[...]
django.utils.datastructures.MultiValueDictKeyError: "'item_text'"

def home_page(request):
    return render(request, 'home.html', {
        'new_item_text': request.POST.get('item_text', ''),
    })

AssertionError: False is not true : New to-do item did not appear in table

self.assertTrue(
    any(row.text == '1: Buy peacock feathers' for row in rows),
    f"New to-do item did not appear in table. Contents were:\n{table.text}"  
)

AssertionError: False is not true : New to-do item did not appear in table.
Contents were:
Buy peacock feathers

    self.assertIn('1: Buy peacock feathers', [row.text for row in rows])

    self.assertIn('1: Buy peacock feathers', [row.text for row in rows])
AssertionError: '1: Buy peacock feathers' not found in ['Buy peacock feathers']

    <tr><td>1: {{ new_item_text }}</td></tr>

    # There is still a text box inviting her to add another item. She
    # enters "Use peacock feathers to make a fly" (Edith is very
    # methodical)
    inputbox = self.browser.find_element_by_id('id_new_item')
    inputbox.send_keys('Use peacock feathers to make a fly')
    inputbox.send_keys(Keys.ENTER)
    time.sleep(1)

    # The page updates again, and now shows both items on her list
    table = self.browser.find_element_by_id('id_list_table')
    rows = table.find_elements_by_tag_name('tr')
    self.assertIn('1: Buy peacock feathers', [row.text for row in rows])
    self.assertIn(
        '2: Use peacock feathers to make a fly',
         [row.text for row in rows]
    )

    # Edith wonders whether the site will remember her list. Then she sees
    # that the site has generated a unique URL for her -- there is some
    # explanatory text to that effect.
    self.fail('Finish the test!')

    # She visits that URL - her to-do list is still there.

AssertionError: '1: Buy peacock feathers' not found in ['1: Use peacock
feathers to make a fly']

Three Strikes and Refactor

Before we go further—we’ve got a bad code smell² in this FT. We have three almost identical code blocks checking for new items in the list table. There’s a principle called Don’t Repeat Yourself (DRY), which we like to apply by following the mantra three strikes and refactor. You can copy and paste code once, and it may be premature to try to remove the duplication it causes, but once you get three occurrences, it’s time to remove duplication.

We start by committing what we have so far. Even though we know our site has a major flaw—it can only handle one list item—it’s still further ahead than it was. We may have to rewrite it all, and we may not, but the rule is that before you do any refactoring, always do a commit:

$ git diff
# should show changes to functional_tests.py, home.html,
# tests.py and views.py
$ git commit -a

Back to our functional test refactor: we could use an inline function, but that upsets the flow of the test slightly. Let’s use a helper method—remember, only methods that begin with test_ will get run as tests, so you can use other methods for your own purposes:

    def tearDown(self):
        self.browser.quit()


    def check_for_row_in_list_table(self, row_text):
        table = self.browser.find_element_by_id('id_list_table')
        rows = table.find_elements_by_tag_name('tr')
        self.assertIn(row_text, [row.text for row in rows])


    def test_can_start_a_list_and_retrieve_it_later(self):
        [...]

I like to put helper methods near the top of the class, between the tearDown and the first test. Let’s use it in the FT:

    # When she hits enter, the page updates, and now the page lists
    # "1: Buy peacock feathers" as an item in a to-do list table
    inputbox.send_keys(Keys.ENTER)
    time.sleep(1)
    self.check_for_row_in_list_table('1: Buy peacock feathers')

    # There is still a text box inviting her to add another item. She
    # enters "Use peacock feathers to make a fly" (Edith is very
    # methodical)
    inputbox = self.browser.find_element_by_id('id_new_item')
    inputbox.send_keys('Use peacock feathers to make a fly')
    inputbox.send_keys(Keys.ENTER)
    time.sleep(1)

    # The page updates again, and now shows both items on her list
    self.check_for_row_in_list_table('1: Buy peacock feathers')
    self.check_for_row_in_list_table('2: Use peacock feathers to make a fly')

    # Edith wonders whether the site will remember her list. Then she sees
    [...]

We run the FT again to check that it still behaves in the same way…

AssertionError: '1: Buy peacock feathers' not found in ['1: Use peacock
feathers to make a fly']

Good. Now we can commit the FT refactor as its own small, atomic change:

$ git diff # check the changes to functional_tests.py
$ git commit -a

And back to work. If we’re ever going to handle more than one list item, we’re going to need some kind of persistence, and databases are a stalwart solution in this area.

The Django ORM and Our First Model

An Object-Relational Mapper (ORM) is a layer of abstraction for data stored in a database with tables, rows, and columns. It lets us work with databases using familiar object-oriented metaphors which work well with code. Classes map to database tables, attributes map to columns, and an individual instance of the class represents a row of data in the database.

Django comes with an excellent ORM, and writing a unit test that uses it is actually an excellent way of learning it, since it exercises code by specifying how we want it to work.

Let’s create a new class in lists/tests.py:

from lists.models import Item
[...]

class ItemModelTest(TestCase):

    def test_saving_and_retrieving_items(self):
        first_item = Item()
        first_item.text = 'The first (ever) list item'
        first_item.save()

        second_item = Item()
        second_item.text = 'Item the second'
        second_item.save()

        saved_items = Item.objects.all()
        self.assertEqual(saved_items.count(), 2)

        first_saved_item = saved_items[0]
        second_saved_item = saved_items[1]
        self.assertEqual(first_saved_item.text, 'The first (ever) list item')
        self.assertEqual(second_saved_item.text, 'Item the second')

You can see that creating a new record in the database is a relatively simple matter of creating an object, assigning some attributes, and calling a .save() function. Django also gives us an API for querying the database via a class attribute, .objects, and we use the simplest possible query, .all(), which retrieves all the records for that table. The results are returned as a list-like object called a QuerySet, from which we can extract individual objects, and also call further functions, like .count(). We then check the objects as saved to the database, to check whether the right information was saved.

Django’s ORM has many other helpful and intuitive features; this might be a good time to skim through the Django tutorial, which has an excellent intro to them.

Let’s try running the unit test. Here comes another unit-test/code cycle:

ImportError: cannot import name 'Item'

Very well, let’s give it something to import from lists/models.py. We’re feeling confident so we’ll skip the Item = None step, and go straight to creating a class:

from django.db import models

class Item(object):
    pass

That gets our test as far as:

    first_item.save()
AttributeError: 'Item' object has no attribute 'save'

To give our Item class a save method, and to make it into a real Django model, we make it inherit from the Model class:

from django.db import models

class Item(models.Model):
    pass

django.db.utils.OperationalError: no such table: lists_item

$ python manage.py makemigrations
Migrations for 'lists':
  lists/migrations/0001_initial.py
    - Create model Item
$ ls lists/migrations
0001_initial.py  __init__.py  __pycache__

$ python manage.py test lists
[...]
    self.assertEqual(first_saved_item.text, 'The first (ever) list item')
AttributeError: 'Item' object has no attribute 'text'

class Item(models.Model):
    text = models.TextField()

django.db.utils.OperationalError: no such column: lists_item.text

$ python manage.py makemigrations
You are trying to add a non-nullable field 'text' to item without a default; we
can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null
value for this column)
 2) Quit, and let me add a default in models.py
Select an option:2

class Item(models.Model):
    text = models.TextField(default='')

$ python manage.py makemigrations
Migrations for 'lists':
  lists/migrations/0002_item_text.py
    - Add field text to item

$ python manage.py test lists
[...]

Ran 3 tests in 0.010s
OK

$ git status # see tests.py, models.py, and 2 untracked migrations
$ git diff # review changes to tests.py and models.py
$ git add lists
$ git commit -m "Model for list Items and associated migration"

Saving the POST to the Database

Let’s adjust the test for our home page POST request, and say we want the view to save a new item to the database instead of just passing it through to its response. We can do that by adding three new lines to the existing test called test_can_save_​a_POST_request:

def test_can_save_a_POST_request(self):
    response = self.client.post('/', data={'item_text': 'A new list item'})

    self.assertEqual(Item.objects.count(), 1)  
    new_item = Item.objects.first()  
    self.assertEqual(new_item.text, 'A new list item')  

    self.assertIn('A new list item', response.content.decode())
    self.assertTemplateUsed(response, 'home.html')

We check that one new Item has been saved to the database. objects.count() is a shorthand for objects.all().count().

objects.first() is the same as doing objects.all()[0].

We check that the item’s text is correct.

This test is getting a little long-winded. It seems to be testing lots of different things. That’s another code smell—a long unit test either needs to be broken into two, or it may be an indication that the thing you’re testing is too complicated. Let’s add that to a little to-do list of our own, perhaps on a piece of scrap paper:

Writing it down on a scratchpad like this reassures us that we won’t forget, so we are comfortable getting back to what we were working on. We rerun the tests and see an expected failure:

    self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1

Let’s adjust our view:

from django.shortcuts import render
from lists.models import Item

def home_page(request):
    item = Item()
    item.text = request.POST.get('item_text', '')
    item.save()

    return render(request, 'home.html', {
        'new_item_text': request.POST.get('item_text', ''),
    })

I’ve coded a very naive solution and you can probably spot a very obvious problem, which is that we’re going to be saving empty items with every request to the home page. Let’s add that to our list of things to fix later. You know, along with the painfully obvious fact that we currently have no way at all of having different lists for different people. That we’ll keep ignoring for now.

Remember, I’m not saying you should always ignore glaring problems like this in “real life”. Whenever we spot problems in advance, there’s a judgement call to make over whether to stop what you’re doing and start again, or leave them until later. Sometimes finishing off what you’re doing is still worth it, and sometimes the problem may be so major as to warrant a stop and rethink.

Let’s see how the unit tests get on…they pass! Good. We can do a bit of refactoring:

    return render(request, 'home.html', {
        'new_item_text': item.text
    })

Let’s have a little look at our scratchpad. I’ve added a couple of the other things that are on our mind:

Let’s start with the first one. We could tack on an assertion to an existing test, but it’s best to keep unit tests to testing one thing at a time, so let’s add a new one:

class HomePageTest(TestCase):
    [...]

    def test_only_saves_items_when_necessary(self):
        self.client.get('/')
        self.assertEqual(Item.objects.count(), 0)

That gives us a 1 != 0 failure. Let’s fix it. Watch out; although it’s quite a small change to the logic of the view, there are quite a few little tweaks to the implementation in code:

def home_page(request):
    if request.method == 'POST':
        new_item_text = request.POST['item_text']  
        Item.objects.create(text=new_item_text)  
    else:
        new_item_text = ''  

    return render(request, 'home.html', {
        'new_item_text': new_item_text,  
    })

We use a variable called new_item_text, which will either hold the POST contents, or the empty string.

.objects.create is a neat shorthand for creating a new Item, without needing to call .save().

And that gets the test passing:

Ran 4 tests in 0.010s

OK

Redirect After a POST

But, yuck, that whole new_item_text = '' dance is making me pretty unhappy. Thankfully we now have an opportunity to fix it. A view function has two jobs: processing user input, and returning an appropriate response. We’ve taken care of the first part, which is saving the users’ input to the database, so now let’s work on the second part.

Always redirect after a POST, they say, so let’s do that. Once again we change our unit test for saving a POST request to say that, instead of rendering a response with the item in it, it should redirect back to the home page:

    def test_can_save_a_POST_request(self):
        response = self.client.post('/', data={'item_text': 'A new list item'})

        self.assertEqual(Item.objects.count(), 1)
        new_item = Item.objects.first()
        self.assertEqual(new_item.text, 'A new list item')

        self.assertEqual(response.status_code, 302)
        self.assertEqual(response['location'], '/')

We no longer expect a response with a .content rendered by a template, so we lose the assertions that look at that. Instead, the response will represent an HTTP redirect, which should have status code 302, and points the browser towards a new location.

That gives us the error 200 != 302. We can now tidy up our view substantially:

from django.shortcuts import redirect, render
from lists.models import Item

def home_page(request):
    if request.method == 'POST':
        Item.objects.create(text=request.POST['item_text'])
        return redirect('/')

    return render(request, 'home.html')

And the tests should now pass:

Ran 4 tests in 0.010s

OK

    def test_can_save_a_POST_request(self):
        self.client.post('/', data={'item_text': 'A new list item'})

        self.assertEqual(Item.objects.count(), 1)
        new_item = Item.objects.first()
        self.assertEqual(new_item.text, 'A new list item')


    def test_redirects_after_POST(self):
        response = self.client.post('/', data={'item_text': 'A new list item'})
        self.assertEqual(response.status_code, 302)
        self.assertEqual(response['location'], '/')

Ran 5 tests in 0.010s

OK

Rendering Items in the Template

Much better! Back to our to-do list:

Crossing things off the list is almost as satisfying as seeing tests pass!

The third item is the last of the “easy” ones. Let’s have a new unit test that checks that the template can also display multiple list items:

class HomePageTest(TestCase):
    [...]

    def test_displays_all_list_items(self):
        Item.objects.create(text='itemey 1')
        Item.objects.create(text='itemey 2')

        response = self.client.get('/')

        self.assertIn('itemey 1', response.content.decode())
        self.assertIn('itemey 2', response.content.decode())

That fails as expected:

AssertionError: 'itemey 1' not found in '<html>\n    <head>\n [...]

The Django template syntax has a tag for iterating through lists, {% for .. in .. %}; we can use it like this:

<table id="id_list_table">
    {% for item in items %}
        <tr><td>1: {{ item.text }}</td></tr>
    {% endfor %}
</table>

This is one of the major strengths of the templating system. Now the template will render with multiple <tr> rows, one for each item in the variable items. Pretty neat! I’ll introduce a few more bits of Django template magic as we go, but at some point you’ll want to go and read up on the rest of them in the Django docs.

Just changing the template doesn’t get our tests to green; we need to actually pass the items to it from our home page view:

def home_page(request):
    if request.method == 'POST':
        Item.objects.create(text=request.POST['item_text'])
        return redirect('/')

    items = Item.objects.all()
    return render(request, 'home.html', {'items': items})

That does get the unit tests to pass…moment of truth, will the functional test pass?

$ python functional_tests.py
[...]
AssertionError: 'To-Do' not found in 'OperationalError at /'

Oops, apparently not. Let’s use another functional test debugging technique, and it’s one of the most straightforward: manually visiting the site! Open up http://localhost:8000 in your web browser, and you’ll see a Django debug page saying “no such table: lists_item”, as in Figure 5-2.

Figure 5-2. Another helpful debug message

Recap

Where are we?

• We’ve got a form set up to add new items to the list using POST.

• We’ve set up a simple model in the database to save list items.

• We’ve learned about creating database migrations, both for the test database (where they’re applied automatically) and for the real database (where we have to apply them manually).

• We’ve used our first couple of Django template tags: {% csrf_token %} and the {% for ... endfor %} loop.

• And we’ve used at least three different FT debugging techniques: in-line print statements, time.sleeps, and improving the error messages.

But we’ve got a couple of items on our own to-do list, namely getting the FT to clean up after itself, and perhaps more critically, adding support for more than one list.

I mean, we could ship the site as it is, but people might find it strange that the entire human population has to share a single to-do list. I suppose it might get people to stop and think about how connected we all are to one another, how we all share a common destiny here on Spaceship Earth, and how we must all work together to solve the global problems that we face.

But in practical terms, the site wouldn’t be very useful.

Ah well.

Ensuring Test Isolation in Functional Tests

We ended the last chapter with a classic testing problem: how to ensure isolation between tests. Each run of our functional tests was leaving list items lying around in the database, and that would interfere with the test results when you next ran the tests.

When we run unit tests, the Django test runner automatically creates a brand new test database (separate from the real one), which it can safely reset before each individual test is run, and then throw away at the end. But our functional tests currently run against the “real” database, db.sqlite3.

One way to tackle this would be to “roll our own” solution, and add some code to functional_tests.py which would do the cleaning up. The setUp and tearDown methods are perfect for this sort of thing.

Since Django 1.4 though, there’s a new class called LiveServerTestCase which can do this work for you. It will automatically create a test database (just like in a unit test run), and start up a development server for the functional tests to run against. Although as a tool it has some limitations which we’ll need to work around later, it’s dead useful at this stage, so let’s check it out.

LiveServerTestCase expects to be run by the Django test runner using manage.py. As of Django 1.6, the test runner will find any files whose name begins with test. To keep things neat and tidy, let’s make a folder for our functional tests, so that it looks a bit like an app. All Django needs is for it to be a valid Python package directory (i.e., one with a ___init___.py in it):

$ mkdir functional_tests
$ touch functional_tests/__init__.py

Then we move our functional tests, from being a standalone file called functional_tests.py, to being the tests.py of the functional_tests app. We use git mv so that Git notices that we’ve moved the file:

$ git mv functional_tests.py functional_tests/tests.py
$ git status # shows the rename to functional_tests/tests.py and __init__.py

At this point your directory tree should look like this:

.
├── db.sqlite3
├── functional_tests
│   ├── __init__.py
│   └── tests.py
├── lists
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   ├── 0001_initial.py
│   │   ├── 0002_item_text.py
│   │   ├── __init__.py
│   │   └── __pycache__
│   ├── models.py
│   ├── __pycache__
│   ├── templates
│   │   └── home.html
│   ├── tests.py
│   └── views.py
├── manage.py
├── superlists
│   ├── __init__.py
│   ├── __pycache__
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── virtualenv
    ├── [...]

functional_tests.py is gone, and has turned into functional_tests/tests.py. Now, whenever we want to run our functional tests, instead of running python functional_tests.py, we will use python manage.py test functional_tests.

Now let’s edit functional_tests/tests.py and change our NewVisitorTest class to make it use LiveServerTestCase:

from django.test import LiveServerTestCase
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time


class NewVisitorTest(LiveServerTestCase):

    def setUp(self):
        [...]

Next, instead of hardcoding the visit to localhost port 8000, LiveServerTestCase gives us an attribute called live_server_url:

    def test_can_start_a_list_and_retrieve_it_later(self):
        # Edith has heard about a cool new online to-do app. She goes
        # to check out its homepage
        self.browser.get(self.live_server_url)

We can also remove the if __name__ == '__main__' from the end if we want, since we’ll be using the Django test runner to launch the FT.

Now we are able to run our functional tests using the Django test runner, by telling it to run just the tests for our new functional_tests app:

$ python manage.py test functional_tests
Creating test database for alias 'default'...
F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later
(functional_tests.tests.NewVisitorTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "...python-tdd-book/functional_tests/tests.py", line 65, in
test_can_start_a_list_and_retrieve_it_later
    self.fail('Finish the test!')
AssertionError: Finish the test!

 ---------------------------------------------------------------------
Ran 1 test in 6.578s

FAILED (failures=1)
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...

The FT gets through to the self.fail, just like it did before the refactor. You’ll also notice that if you run the tests a second time, there aren’t any old list items lying around from the previous test—it has cleaned up after itself. Success! We should commit it as an atomic change:

$ git status # functional_tests.py renamed + modified, new __init__.py
$ git add functional_tests
$ git diff --staged -M
$ git commit  # msg eg "make functional_tests an app, use LiveServerTestCase"

The -M flag on the git diff is a useful one. It means “detect moves”, so it will notice that functional_tests.py and functional_tests/tests.py are the same file, and show you a more sensible diff (try it without the flag!).

$ python manage.py test
Creating test database for alias 'default'...
......F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later
[...]
AssertionError: Finish the test!

 ---------------------------------------------------------------------
Ran 7 tests in 6.732s

FAILED (failures=1)

$ python manage.py test lists
Creating test database for alias 'default'...
......
 ---------------------------------------------------------------------
Ran 6 tests in 0.009s

OK
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...

Aside: Upgrading Selenium and Geckodriver

As I was running through this chapter again today, I found the FTs hung when I tried to run them.

It turns out that Firefox had auto-updated itself overnight, and my versions of Selenium and Geckodriver needed upgrading too. A quick visit to the geckodriver releases page confirmed there was a new version out. So a few downloads and upgrades were in order:

• A quick pip install --upgrade selenium first.

• Then a quick download of the new geckodriver.

• I saved a backup copy of the old one somewhere, and put the new one in its place somewhere on the PATH.

• And a quick check with geckodriver --version confirms the new one was ready to go.

The FTs were then back to running the way I expected them to.

There was no particular reason that it happened at this point in the book; indeed, it’s quite unlikely that it’ll happen right now for you, but it may happen at some point, and this seemed as good a place as any to talk about it, since we’re doing some housekeeping.

It’s one of the things you have to put up with when using Selenium. Although it is possible to pin your browser and Selenium versions (on a CI server, for example), browser versions don’t stand still out in the real world, and you need to keep up with what your users have.

Back to our regular programming now.

On Implicit and Explicit Waits, and Voodoo time.sleeps

Let’s talk about the time.sleep in our FT:

        # When she hits enter, the page updates, and now the page lists
        # "1: Buy peacock feathers" as an item in a to-do list table
        inputbox.send_keys(Keys.ENTER)
        time.sleep(1)

        self.check_for_row_in_list_table('1: Buy peacock feathers')

This is what’s called an “explicit wait”. That’s by contrast with “implicit waits”: in certain cases, Selenium tries to wait “automatically” for you when it thinks the page is loading. It even provides a method called implicitly_wait that lets you control how long it will wait if you ask it for an element that doesn’t seem to be on the page yet.

In fact, in the first edition, I was able to rely entirely on implicit waits. The problem is that implicit waits are always a little flakey, and with the release of Selenium 3, implicit waits became even more unreliable. At the same time, the general opinion from the Selenium team was that implicit waits were just a bad idea, and to be avoided.

So this edition has explicit waits from the very beginning. But the problem is that those time.sleeps have their own issues. Currently we’re waiting for one second, but who’s to say that’s the right amount of time? For most tests we run against our own machine, one second is way too long, and it’s going to really slow down our FT runs. 0.1s would be fine. But the problem is that if you set it that low, every so often you’re going to get a spurious failure because, for whatever reason, the laptop was being a bit slow just then. And even at 1s you can never be quite sure you’re not going to get random failures that don’t indicate a real problem, and false positives in tests are a real annoyance (there’s lots more on this in an article by Martin Fowler).

So let’s replace our sleeps with a tool that will wait for just as long as is needed, up to a nice long timeout to catch any glitches. We’ll rename check_for_row_in_list_table to wait_for_row_in_list_table, and add some polling/retry logic to it:

from selenium.common.exceptions import WebDriverException

MAX_WAIT = 10  
[...]

    def wait_for_row_in_list_table(self, row_text):
        start_time = time.time()
        while True:  
            try:
                table = self.browser.find_element_by_id('id_list_table')  
                rows = table.find_elements_by_tag_name('tr')
                self.assertIn(row_text, [row.text for row in rows])
                return  
            except (AssertionError, WebDriverException) as e:  
                if time.time() - start_time > MAX_WAIT:  
                    raise e  
                time.sleep(0.5)  

We’ll use a constant called MAX_WAIT to set the maximum amount of time we’re prepared to wait. 10 seconds should be more than enough to catch any glitches or random slowness.

Here’s the loop, which will keep going forever, unless we get to one of two possible exit routes.

Here are our three lines of assertions from the old version of the method.

If we get through them and our assertion passes, we return from the function and escape the loop.

But if we catch an exception, we wait a short amount of time and loop around to retry. There are two types of exceptions we want to catch: WebDriverException for when the page hasn’t loaded and Selenium can’t find the table element on the page, and AssertionError for when the table is there, but it’s perhaps a table from before the page reloads, so it doesn’t have our row in yet.

Here’s our second escape route. If we get to this point, that means our code kept raising exceptions every time we tried it until we exceeded our timeout. So this time, we re-raise the exception and let it bubble up to our test, and most likely end up in our traceback, telling us why the test failed.

Are you thinking this code is a little ugly, and makes it a bit harder to see exactly what we’re doing? I agree. Later on, we’ll refactor out a general wait_for helper, to separate the timing and re-raising logic from the test assertions. But we’ll wait until we need it in multiple places.

Now we can rename our method calls, and remove the voodoo time.sleeps:

    [...]
    # When she hits enter, the page updates, and now the page lists
    # "1: Buy peacock feathers" as an item in a to-do list table
    inputbox.send_keys(Keys.ENTER)
    self.wait_for_row_in_list_table('1: Buy peacock feathers')

    # There is still a text box inviting her to add another item. She
    # enters "Use peacock feathers to make a fly" (Edith is very
    # methodical)
    inputbox = self.browser.find_element_by_id('id_new_item')
    inputbox.send_keys('Use peacock feathers to make a fly')
    inputbox.send_keys(Keys.ENTER)

    # The page updates again, and now shows both items on her list
    self.wait_for_row_in_list_table('2: Use peacock feathers to make a fly')
    self.wait_for_row_in_list_table('1: Buy peacock feathers')
    [...]

And rerun the tests:

$ python manage.py test
Creating test database for alias 'default'...
......F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later
(functional_tests.tests.NewVisitorTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "...python-tdd-book/functional_tests/tests.py", line 73, in
test_can_start_a_list_and_retrieve_it_later
    self.fail('Finish the test!')
AssertionError: Finish the test!

 ---------------------------------------------------------------------
Ran 7 tests in 4.552s

FAILED (failures=1)
System check identified no issues (0 silenced).
Destroying test database for alias 'default'...

We get to the same place, and notice we’ve shaved a couple of seconds off the execution time too. That might not seem like a lot right now, but it all adds up.

Just to check we’ve done the right thing, let’s deliberately break the test in a couple of ways and see some errors. First let’s check that if we look for some row text that will never appear, we get the right error:

        rows = table.find_elements_by_tag_name('tr')
        self.assertIn('foo', [row.text for row in rows])
        return

We see we still get a nice self-explanatory test failure message:

    self.assertIn('foo', [row.text for row in rows])
AssertionError: 'foo' not found in ['1: Buy peacock feathers']

Let’s put that back the way it was and break something else:

    try:
        table = self.browser.find_element_by_id('id_nothing')
        rows = table.find_elements_by_tag_name('tr')
        self.assertIn(row_text, [row.text for row in rows])
        return
    [...]

Sure enough, we get the errors for when the page doesn’t contain the element we’re looking for too:

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_nothing"]

Everything seems to be in order. Let’s put our code back to way it should be, and do one final test run:

$ python manage.py test
[...]
AssertionError: Finish the test!

Great. With that little interlude over, let’s crack on with getting our application actually working for multiple lists.

Small Design When Necessary

Let’s have a think about how we want support for multiple lists to work. Currently the FT (which is the closest we have to a design document) says this:

    # Edith wonders whether the site will remember her list. Then she sees
    # that the site has generated a unique URL for her -- there is some
    # explanatory text to that effect.
    self.fail('Finish the test!')

    # She visits that URL - her to-do list is still there.

    # Satisfied, she goes back to sleep

But really we want to expand on this, by saying that different users don’t see each other’s lists, and each get their own URL as a way of going back to their saved lists. What might a new design look like?

    /lists/<list identifier>/

    /lists/new

    /lists/<list identifier>/add_item

Implementing the New Design Incrementally Using TDD

How do we use TDD to implement the new design? Let’s take another look at the flowchart for the TDD process in Figure 7-1.

At the top level, we’re going to use a combination of adding new functionality (by adding a new FT and writing new application code), and refactoring our application—that is, rewriting some of the existing implementation so that it delivers the same functionality to the user but using aspects of our new design. We’ll be able to use the existing functional test to verify we don’t break what already works, and the new functional test to drive the new features.

At the unit test level, we’ll be adding new tests or modifying existing ones to test for the changes we want, and we’ll be able to similarly use the unit tests we don’t touch to help make sure we don’t break anything in the process.

Figure 7-1. The TDD process with functional and unit tests

Iterating Towards the New Design

Being all excited about our new design, I had an overwhelming urge to dive in at this point and start changing models.py, which would have broken half the unit tests, and then pile in and change almost every single line of code, all in one go. That’s a natural urge, and TDD, as a discipline, is a constant fight against it. Obey the Testing Goat, not Refactoring Cat! We don’t need to implement our new, shiny design in a single big bang. Let’s make small changes that take us from a working state to a working state, with our design guiding us gently at each stage.

There are four items on our to-do list. The FT, with its Regexp didn't match, is telling us that the second item—giving lists their own URL and identifier—is the one we should work on next. Let’s have a go at fixing that, and only that.

The URL comes from the redirect after POST. In lists/tests.py, find test_redirects_after_POST, and change the expected redirect location:

self.assertEqual(response.status_code, 302)
self.assertEqual(response['location'], '/lists/the-only-list-in-the-world/')

Does that seem slightly strange? Clearly, /lists/the-only-list-in-the-world isn’t a URL that’s going to feature in the final design of our application. But we’re committed to changing one thing at a time. While our application only supports one list, this is the only URL that makes sense. We’re still moving forwards, in that we’ll have a different URL for our list and our home page, which is a step along the way to a more REST-ful design. Later, when we have multiple lists, it will be easy to change.

Running the unit tests gives us an expected fail:

$ python manage.py test lists
[...]
AssertionError: '/' != '/lists/the-only-list-in-the-world/'

We can go adjust our home_page view in lists/views.py:

def home_page(request):
    if request.method == 'POST':
        Item.objects.create(text=request.POST['item_text'])
        return redirect('/lists/the-only-list-in-the-world/')

    items = Item.objects.all()
    return render(request, 'home.html', {'items': items})

Of course, that will now totally break the functional tests, because there is no such URL on our site yet. Sure enough, if you run them, you’ll find they fail just after trying to submit the first item, saying that they can’t find the list table; it’s because the URL /the-only-list-in-the-world/ doesn’t exist yet!

  File "...python-tdd-book/functional_tests/tests.py", line 57, in
test_can_start_a_list_for_one_user
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]

[...]

  File "...python-tdd-book/functional_tests/tests.py", line 79, in
test_multiple_users_can_start_lists_at_different_urls
    self.wait_for_row_in_list_table('1: Buy peacock feathers')
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]

Not only is our new test failing, but the old one is too. That tells us we’ve introduced a regression. Let’s try to get back to a working state as quickly as possible by building a URL for our one and only list.

Taking a First, Self-Contained Step: One New URL

Open up lists/tests.py, and add a new test class called ListViewTest. Then copy the method called test_displays_all_list_items across from HomePageTest into our new class, rename it, and adapt it slightly:

class ListViewTest(TestCase):

    def test_displays_all_items(self):
        Item.objects.create(text='itemey 1')
        Item.objects.create(text='itemey 2')

        response = self.client.get('/lists/the-only-list-in-the-world/')

        self.assertContains(response, 'itemey 1')  
        self.assertContains(response, 'itemey 2')  

Here’s a new helper method: instead of using the slightly annoying assertIn/response.content.decode() dance, Django provides the assertContains method, which knows how to deal with responses and the bytes of their content.

Let’s try running this test now:

    self.assertContains(response, 'itemey 1')
[...]
AssertionError: 404 != 200 : Couldn't retrieve content: Response code was 404

Here’s a nice side effect of using assertContains: it tells us straight away that the test is failing because our new URL doesn’t exist yet, and is returning a 404.

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
    url(r'^lists/the-only-list-in-the-world/$', views.view_list, name='view_list'),
]

AttributeError: module 'lists.views' has no attribute 'view_list'

def view_list(request):
    pass

ValueError: The view lists.views.view_list didn't return an HttpResponse
object. It returned None instead.

[...]
FAILED (errors=1)

def view_list(request):
    items = Item.objects.all()
    return render(request, 'home.html', {'items': items})

Ran 7 tests in 0.016s
OK

FAIL: test_can_start_a_list_for_one_user
[...]
  File "...python-tdd-book/functional_tests/tests.py", line 67, in
test_can_start_a_list_for_one_user
[...]
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy
peacock feathers']

FAIL: test_multiple_users_can_start_lists_at_different_urls
[...]
AssertionError: 'Buy peacock feathers' unexpectedly found in 'Your To-Do
list\n1: Buy peacock feathers'
[...]

        <form method="POST">

        <form method="POST" action="/">

FAIL: test_multiple_users_can_start_lists_at_different_urls
[...]
AssertionError: 'Buy peacock feathers' unexpectedly found in 'Your To-Do
list\n1: Buy peacock feathers'

Ran 2 tests in 8.541s
FAILED (failures=1)

Green? Refactor

Time for a little tidying up.

In the Red/Green/Refactor dance, we’ve arrived at green, so we should see what needs a refactor. We now have two views, one for the home page, and one for an individual list. Both are currently using the same template, and passing it all the list items currently in the database. If we look through our unit test methods, we can see some stuff we probably want to change:

$ grep -E "class|def" lists/tests.py
class HomePageTest(TestCase):
    def test_uses_home_template(self):
    def test_displays_all_list_items(self):
    def test_can_save_a_POST_request(self):
    def test_redirects_after_POST(self):
    def test_only_saves_items_when_necessary(self):
class ListViewTest(TestCase):
    def test_displays_all_items(self):
class ItemModelTest(TestCase):
    def test_saving_and_retrieving_items(self):

We can definitely delete the test_displays_all_list_items method from HomePageTest; it’s no longer needed. If you run manage.py test lists now, it should say it ran 6 tests instead of 7:

Ran 6 tests in 0.016s
OK

Next, since we don’t actually need the home page template to display all list items any more, it should just show a single input box inviting you to start a new list.

Another Small Step: A Separate Template for Viewing Lists

Since the home page and the list view are now quite distinct pages, they should be using different HTML templates; home.html can have the single input box, whereas a new template, list.html, can take care of showing the table of existing items.

Let’s add a new test to check that it’s using a different template:

class ListViewTest(TestCase):

    def test_uses_list_template(self):
        response = self.client.get('/lists/the-only-list-in-the-world/')
        self.assertTemplateUsed(response, 'list.html')


    def test_displays_all_items(self):
        [...]

assertTemplateUsed is one of the more useful functions that the Django Test Client gives us. Let’s see what it says:

AssertionError: False is not true : Template 'list.html' was not a template
used to render the response. Actual template(s) used: home.html

Great! Let’s change the view:

def view_list(request):
    items = Item.objects.all()
    return render(request, 'list.html', {'items': items})

But, obviously, that template doesn’t exist yet. If we run the unit tests, we get:

django.template.exceptions.TemplateDoesNotExist: list.html

Let’s create a new file at lists/templates/list.html:

$ touch lists/templates/list.html

A blank template, which gives us this error—good to know the tests are there to make sure we fill it in:

AssertionError: False is not true : Couldn't find 'itemey 1' in response

The template for an individual list will reuse quite a lot of the stuff we currently have in home.html, so we can start by just copying that:

$ cp lists/templates/home.html lists/templates/list.html

That gets the tests back to passing (green). Now let’s do a little more tidying up (refactoring). We said the home page doesn’t need to list items, it only needs the new list input field, so we can remove some lines from lists/templates/home.html, and maybe slightly tweak the h1 to say “Start a new To-Do list”:

<body>
  <h1>Start a new To-Do list</h1>
  <form method="POST">
    <input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
    {% csrf_token %}
  </form>
</body>

We rerun the unit tests to check that hasn’t broken anything—good…

There’s actually no need to pass all the items to the home.html template in our home_page view, so we can simplify that:

def home_page(request):
    if request.method == 'POST':
        Item.objects.create(text=request.POST['item_text'])
        return redirect('/lists/the-only-list-in-the-world/')
    return render(request, 'home.html')

Rerun the unit tests once more; they still pass. Time to run the functional tests:

AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy
milk']

Not bad! Our regression test (the first FT) is passing, and our new test is now getting slightly further forwards—it’s telling us that Francis isn’t getting his own list page (because he still sees some of Edith’s list items).

It may feel like we haven’t made much headway since, functionally, the site still behaves almost exactly like it did when we started the chapter, but this really is progress. We’ve started on the road to our new design, and we’ve implemented a number of stepping stones without making anything worse than it was before. Let’s commit our progress so far:

$ git status # should show 4 changed files and 1 new file, list.html
$ git add lists/templates/list.html
$ git diff # should show we've simplified home.html,
           # moved one test to a new class in lists/tests.py added a new view
           # in views.py, and simplified home_page and made one addition to
           # urls.py
$ git commit -a # add a message summarising the above, maybe something like
                # "new URL, view and template to display lists"

A Third Small Step: A URL for Adding List Items

Where are we with our own to-do list?

We’ve sort of made progress on the second item, even if there’s still only one list in the world. The first item is a bit scary. Can we do something about items 3 or 4?

Let’s have a new URL for adding new list items. If nothing else, it’ll simplify the home page view.

class NewListTest(TestCase):

    def test_can_save_a_POST_request(self):
        self.client.post('/lists/new', data={'item_text': 'A new list item'})
        self.assertEqual(Item.objects.count(), 1)
        new_item = Item.objects.first()
        self.assertEqual(new_item.text, 'A new list item')


    def test_redirects_after_POST(self):
        response = self.client.post('/lists/new', data={'item_text': 'A new list item'})
        self.assertEqual(response.status_code, 302)
        self.assertEqual(response['location'], '/lists/the-only-list-in-the-world/')

    def test_redirects_after_POST(self):
        response = self.client.post('/lists/new', data={'item_text': 'A new list item'})
        self.assertRedirects(response, '/lists/the-only-list-in-the-world/')

    self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1
[...]
    self.assertRedirects(response, '/lists/the-only-list-in-the-world/')
[...]
AssertionError: 404 != 302 : Response didn't redirect as expected: Response
code was 404 (expected 302)

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
    url(r'^lists/new$', views.new_list, name='new_list'),
    url(r'^lists/the-only-list-in-the-world/$', views.view_list, name='view_list'),
]

def new_list(request):
    pass

def new_list(request):
    return redirect('/lists/the-only-list-in-the-world/')

    self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1

def new_list(request):
    Item.objects.create(text=request.POST['item_text'])
    return redirect('/lists/the-only-list-in-the-world/')

Ran 7 tests in 0.030s

OK

[...]
AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy
milk']
Ran 2 tests in 8.972s
FAILED (failures=1)

def home_page(request):
    return render(request, 'home.html')

OK

Ran 6 tests in 0.016s
OK

ERROR: test_can_start_a_list_for_one_user
[...]
  File "...python-tdd-book/functional_tests/tests.py", line 57, in
test_can_start_a_list_for_one_user
    self.wait_for_row_in_list_table('1: Buy peacock feathers')
  File "...python-tdd-book/functional_tests/tests.py", line 23, in
wait_for_row_in_list_table
    table = self.browser.find_element_by_id('id_list_table')
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]

ERROR: test_multiple_users_can_start_lists_at_different_urls
[...]
  File "...python-tdd-book/functional_tests/tests.py", line 79, in
test_multiple_users_can_start_lists_at_different_urls
    self.wait_for_row_in_list_table('1: Buy peacock feathers')
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]
[...]

Ran 2 tests in 11.592s
FAILED (errors=2)

    <form method="POST" action="/lists/new">

AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy
milk']
[...]
FAILED (failures=1)

$ git status # 5 changed files
$ git diff # URLs for forms x2, moved code in views + tests, new URL
$ git commit -a

Biting the Bullet: Adjusting Our Models

Enough housekeeping with our URLs. It’s time to bite the bullet and change our models. Let’s adjust the model unit test. Just for a change, I’ll present the changes in the form of a diff:

@@ -1,5 +1,5 @@
 from django.test import TestCase
-from lists.models import Item
+from lists.models import Item, List


 class HomePageTest(TestCase):
@@ -44,22 +44,32 @@ class ListViewTest(TestCase):



-class ItemModelTest(TestCase):
+class ListAndItemModelsTest(TestCase):

     def test_saving_and_retrieving_items(self):
+        list_ = List()
+        list_.save()
+
         first_item = Item()
         first_item.text = 'The first (ever) list item'
+        first_item.list = list_
         first_item.save()

         second_item = Item()
         second_item.text = 'Item the second'
+        second_item.list = list_
         second_item.save()

+        saved_list = List.objects.first()
+        self.assertEqual(saved_list, list_)
+
         saved_items = Item.objects.all()
         self.assertEqual(saved_items.count(), 2)

         first_saved_item = saved_items[0]
         second_saved_item = saved_items[1]
         self.assertEqual(first_saved_item.text, 'The first (ever) list item')
+        self.assertEqual(first_saved_item.list, list_)
         self.assertEqual(second_saved_item.text, 'Item the second')
+        self.assertEqual(second_saved_item.list, list_)

We create a new List object, and then we assign each item to it by assigning it as its .list property. We check that the list is properly saved, and we check that the two items have also saved their relationship to the list. You’ll also notice that we can compare list objects with each other directly (saved_list and list_)—behind the scenes, these will compare themselves by checking that their primary key (the .id attribute) is the same.

Time for another unit-test/code cycle.

For the first couple of iterations, rather than explicitly showing you what code to enter in between every test run, I’m only going to show you the expected error messages from running the tests. I’ll let you figure out what each minimal code change should be on your own.

Your first error should be:

ImportError: cannot import name 'List'

Fix that, and then you should see:

AttributeError: 'List' object has no attribute 'save'

Next you should see:

django.db.utils.OperationalError: no such table: lists_list

So we run a makemigrations:

$ python manage.py makemigrations
Migrations for 'lists':
  lists/migrations/0003_list.py
    - Create model List

And then you should see:

    self.assertEqual(first_saved_item.list, list_)
AttributeError: 'Item' object has no attribute 'list'

from django.db import models

class List(models.Model):
    pass

class Item(models.Model):
    text = models.TextField(default='')
    list = models.TextField(default='')

$ python manage.py test lists
[...]
django.db.utils.OperationalError: no such column: lists_item.list

$ python manage.py makemigrations
Migrations for 'lists':
  lists/migrations/0004_item_list.py
    - Add field list to item

AssertionError: 'List object' != <List: List object>

from django.db import models

class List(models.Model):
    pass


class Item(models.Model):
    text = models.TextField(default='')
    list = models.ForeignKey(List, default=None)

$ rm lists/migrations/0004_item_list.py
$ python manage.py makemigrations
Migrations for 'lists':
  lists/migrations/0004_item_list.py
    - Add field list to item

$ python manage.py test lists
[...]
ERROR: test_displays_all_items (lists.tests.ListViewTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id
[...]
ERROR: test_redirects_after_POST (lists.tests.NewListTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id
[...]
ERROR: test_can_save_a_POST_request (lists.tests.NewListTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id

Ran 6 tests in 0.021s

FAILED (errors=3)

class ListViewTest(TestCase):

    def test_displays_all_items(self):
        list_ = List.objects.create()
        Item.objects.create(text='itemey 1', list=list_)
        Item.objects.create(text='itemey 2', list=list_)

File "...python-tdd-book/lists/views.py", line 9, in new_list
Item.objects.create(text=request.POST['item_text'])

from lists.models import Item, List
[...]
def new_list(request):
    list_ = List.objects.create()
    Item.objects.create(text=request.POST['item_text'], list=list_)
    return redirect('/lists/the-only-list-in-the-world/')

Ran 6 tests in 0.030s

OK

AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy
milk']
[...]

$ git status # 3 changed files, plus 2 migrations
$ git add lists
$ git diff --staged
$ git commit

Each List Should Have Its Own URL

What shall we use as the unique identifier for our lists? Probably the simplest thing, for now, is just to use the auto-generated id field from the database. Let’s change ListViewTest so that the two tests point at new URLs.

We’ll also change the old test_displays_all_items test and call it test_displays_only_items_for_that_list instead, and make it check that only the items for a specific list are displayed:

class ListViewTest(TestCase):

    def test_uses_list_template(self):
        list_ = List.objects.create()
        response = self.client.get(f'/lists/{list_.id}/')
        self.assertTemplateUsed(response, 'list.html')


    def test_displays_only_items_for_that_list(self):
        correct_list = List.objects.create()
        Item.objects.create(text='itemey 1', list=correct_list)
        Item.objects.create(text='itemey 2', list=correct_list)
        other_list = List.objects.create()
        Item.objects.create(text='other list item 1', list=other_list)
        Item.objects.create(text='other list item 2', list=other_list)

        response = self.client.get(f'/lists/{correct_list.id}/')

        self.assertContains(response, 'itemey 1')
        self.assertContains(response, 'itemey 2')
        self.assertNotContains(response, 'other list item 1')
        self.assertNotContains(response, 'other list item 2')

Running the unit tests gives an expected 404, and another related error:

FAIL: test_displays_only_items_for_that_list (lists.tests.ListViewTest)
AssertionError: 404 != 200 : Couldn't retrieve content: Response code was 404
(expected 200)
[...]
FAIL: test_uses_list_template (lists.tests.ListViewTest)
AssertionError: No templates used to render the response

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
    url(r'^lists/new$', views.new_list, name='new_list'),
    url(r'^lists/(.+)/$', views.view_list, name='view_list'),
]

ERROR: test_displays_only_items_for_that_list (lists.tests.ListViewTest)
[...]
TypeError: view_list() takes 1 positional argument but 2 were given
[...]
ERROR: test_uses_list_template (lists.tests.ListViewTest)
[...]
TypeError: view_list() takes 1 positional argument but 2 were given
[...]
ERROR: test_redirects_after_POST (lists.tests.NewListTest)
[...]
TypeError: view_list() takes 1 positional argument but 2 were given
FAILED (errors=3)

def view_list(request, list_id):
    [...]

FAIL: test_displays_only_items_for_that_list (lists.tests.ListViewTest)
[...]
AssertionError: 1 != 0 : Response should not contain 'other list item 1'

def view_list(request, list_id):
    list_ = List.objects.get(id=list_id)
    items = Item.objects.filter(list=list_)
    return render(request, 'list.html', {'items': items})

ERROR: test_redirects_after_POST (lists.tests.NewListTest)
ValueError: invalid literal for int() with base 10:
'the-only-list-in-the-world'

class NewListTest(TestCase):
    [...]

    def test_redirects_after_POST(self):
        response = self.client.post('/lists/new', data={'item_text': 'A new list item'})
        self.assertRedirects(response, '/lists/the-only-list-in-the-world/')

    def test_redirects_after_POST(self):
        response = self.client.post('/lists/new', data={'item_text': 'A new list item'})
        new_list = List.objects.first()
        self.assertRedirects(response, f'/lists/{new_list.id}/')

def new_list(request):
    list_ = List.objects.create()
    Item.objects.create(text=request.POST['item_text'], list=list_)
    return redirect(f'/lists/{list_.id}/')

$ python3 manage.py test lists
[...]
......
 ---------------------------------------------------------------------
Ran 6 tests in 0.033s

OK

One More View to Handle Adding Items to an Existing List

We need a URL and view to handle adding a new item to an existing list (/lists/<list_id>/add_item). We’re getting pretty good at these now, so let’s knock one together quickly:

class NewItemTest(TestCase):

    def test_can_save_a_POST_request_to_an_existing_list(self):
        other_list = List.objects.create()
        correct_list = List.objects.create()

        self.client.post(
            f'/lists/{correct_list.id}/add_item',
            data={'item_text': 'A new item for an existing list'}
        )

        self.assertEqual(Item.objects.count(), 1)
        new_item = Item.objects.first()
        self.assertEqual(new_item.text, 'A new item for an existing list')
        self.assertEqual(new_item.list, correct_list)


    def test_redirects_to_list_view(self):
        other_list = List.objects.create()
        correct_list = List.objects.create()

        response = self.client.post(
            f'/lists/{correct_list.id}/add_item',
            data={'item_text': 'A new item for an existing list'}
        )

        self.assertRedirects(response, f'/lists/{correct_list.id}/')