Maintainable JavaScript

Chapter 19. Automated Testing

Testing JavaScript has long been a pain point for developers. You want to test JavaScript quickly and easily, but there are so many browsers to test. The first solution was manual testing across all browsers, which meant creating an HTML file and manually loading it in various browsers to ensure that it worked. Though functional, this approach was too slow for practical use.

The next wave of JavaScript testing focused on command-line testing by stubbing out the browser environment. Several attempts were made to get JavaScript testing on the command line using Rhino and a fake browser environment. Some companies even developed browser profiles that could be loaded in with the promise of cross-browser testing. The unfortunate reality was that these stubbed browser environments didn’t do the job. Trying to recreate a truly unique environment by hand led to inconsistencies: your tests might pass in the “fake Firefox” but fail in the actual browser.

More recently, attempts have been made to use the actual browsers for testing. This approach typically involves using an HTML file to launch tests and then having an application load that file in the different browsers. Many of the tools mentioned in this chapter are still under development, but they all give you a good starting point for integrating browser-based JavaScript testing.

YUI Test Selenium Driver

YUI Test is the unit testing framework for the YUI Library. The most recent version of YUI Test is more than a simple testing library. In addition to removing dependencies on the core YUI library, YUI Test supports a suite of utilities to aid JavaScript testing. One of these tools is called the YUI Test Selenium Driver and is designed to work with Selenium to enable easy browser testing.

Selenium is a server that is capable of launching browsers and running commands inside of them. Originally intended for use by QA engineers writing functional tests, Selenium gained popularity for JavaScript testing due to the ease with which it interacts with browsers.

Setting Up a Selenium Server

The YUI Test Selenium Driver works with a Selenium server to run JavaScript tests on various browsers and return the results. The first step is to set up your Selenium server (if you don’t already have one). Selenium is written in Java, so it can be run anywhere Java is installed. Download the latest Selenium server from http://seleniumhq.org/download/.

To run your Selenium server, go to the directory containing the downloaded files and run:

java -jar selenium-server-standalone-x.y.z.jar

The server takes a few moments to set up and then is ready to receive commands.

Setting Up YUI Test Selenium Driver

There are three steps to setting up the YUI Test Selenium Driver:

Download the latest version of YUI Test.
Place yuitest-selenium-driver.jar in your dependencies directory.
Copy selenium-java-client-driver.jar from the YUI Test lib directory into your Java Runtime Environment (JRE)’s lib/ext directory.

With these steps completed, it’s now possible to run tests using the YUI Test Selenium Driver.

Using the YUI Test Selenium Driver

The YUI Test Selenium Driver uses HTML files for testing. Even if your tests are in standalone JavaScript files, you must include them in an HTML file that automatically runs the tests upon page load. The following is an example test page:

<!DOCTYPE html>
<html>
<head>
    <title>YUI Test</title>

    <!-- include YUI Test library -->
    <script src="yuitest.js"></script>

    <!-- include your test files -->
    <script src="tests1.js"></script>
    <script src="tests2.js"></script>
</head>
<body>
    <script>
        YUITest.TestRunner.run();
    </script>
</body>
</html>

Each of the JavaScript test files should add their tests to YUI Test via YUITest.TestRunner.add(). That way, the tests can simply be run automatically once they’re fully loaded.

Assuming this HTML file lives on a server as http://www.example.com/tests.html, you can then run the tests using the following command:

java -jar yuitest-selenium-driver.jar [options] [url]+

For example:

java -jar yuitest-selenium-driver.jar http://www.example.com/tests.html

This command runs the given file in Firefox (the default browser on Selenium) and assumes the Selenium server is running on localhost:4444 (the default Selenium port). You can change the location of Selenium by adding some options:

java -jar yuitest-selenium-driver.jar --host testing.example.com 
    --port 9000 http://www.example.com/tests.html

This command runs the given file in Firefox on the Selenium server at testing.example.com:9000. You can also specify additional browsers using the Selenium IDs:

java -jar yuitest-selenium-driver.jar 
    --browsers *firefox,*iexplore http://www.example.com/tests.html

This command runs the tests both in Firefox and in Internet Explorer. The --browsers option passes through these options directly to Selenium and must therefore specify valid Selenium browsers. An error occurs if a given browser isn’t available on the specified Selenium server.

Although it’s possible to pass test URLs on the command line, most developers use the test configuration XML file instead. The XML file has the following format:

<?xml version="1.0"?>
<yuitest>
    <tests base="http://www.example.com/tests/" timeout="10000">
        <url>test_core</url>
        <url timeout="30000">test_util</url>
        <url>test_ui</url>
    </tests>
</yuitest>

The <tests> element is used to specify a base path and default timeout value for each test. Then, each <url> element specifies the relative path to a test page. Tell the YUI Test Selenium Driver to use the XML file instead of the command line by specifying the --tests option:

java -jar yuitest-selenium-driver.jar --tests tests.xml

The YUI Test Selenium Driver then goes through each of the tests, runs them on each of the specified browsers, and returns all of the results.

There is an --erroronfail option that indicates the YUI Test Selenium Driver should exit with a nonzero code when a test fails. It’s a good idea to specify this option so that the build will stop when a test fails rather than continue.

The Ant Target

To create the Ant target, start by specifying the key pieces of data in a properties file:

src.dir = ./src
lib.dir = ./lib
tests.dir = ./tests

yuitestselenium = ${lib.dir}/yuitest-selenium-driver.jar

yuitestselenium.host = testing.example.com
yuitestselenium.port = 4444
yuitestselenium.tests = ${tests.dir}/tests.xml
yuitestselenium.browsers = *firefox

The Ant target uses the <exec> task to run the YUI Test Selenium Driver and pass in the relevant information:

<target name="test">

    <exec executable="java" failonerror="true">
        <arg line="-jar"/>
        <arg path="${yuitestselenium}"/>
        <arg line="--host ${yuitestselenium.host}"/>
        <arg line="--port ${yuitestselenium.port}"/>
        <arg line="--browsers ${yuitestselenium.browsers}"/>
        <arg line="--tests ${yuitestselenium.tests}"/>
        <arg line="--erroronfail"/>
    </exec>     

</target>

The test target will fail if the Selenium Server isn’t running, or if a test times out, or if a specified browser doesn’t exist. It’s important to check all of these conditions before running tests.

The Buildr <yuitest-selenium> task encapsulates all of this functionality. The -erroronfail flag is always passed in, and the other options are available as attributes:

<target name="test">

    <yuitest-selenium host="${yuitestselenium.host}" 
        port="${yuitestselenium.port}" browsers="${yuitestselenium.browsers}"
        tests="${yuitestselenium.tests}"/>

</target>

The tests attribute is required for <yuitest-selenium>; all other attributes are optional.

Yeti

Yeti is another tool designed to work with YUI Test. Unlike the YUI Test Selenium Driver, Yeti is a completely standalone solution written in JavaScript that runs on Node.js. Yeti requires you to have an HTML file that automatically executes your tests, so you can use the same HTML files that are used with the YUI Test Selenium Driver.

You can install Yeti via npm with:

sudo npm install -g yeti

Running Yeti is simply a matter of passing in the HTML file on the command line:

yeti test.html

This command runs all of the tests on Firefox or Safari by default (depending on platform) and outputs the result to the command line. If a Yeti server is running locally, then this command also runs tests on all connected browsers and reports all results.

Due to the simplicity of Yeti, the Ant target requires very little configuration in a properties file:

src.dir = ./src
lib.dir = ./lib
tests.dir = ./tests

yeti = yeti

And the Ant target itself is very straightforward as well:

<target name="test">

    <apply executable="yeti" failonerror="true" parallel="true">
        <fileset dir="${tests.dir}" includes="**/*.html" />
        <srcfile/>
    </apply>

</target>

The test target uses the <apply> task to pass all HTML files found in the tests directory to Yeti. The results are output on the screen, and the build will fail if there are errors.

The Buildr <yeti> task makes using Yeti even simpler:

<target name="test">

    <yeti>
        <fileset dir="${tests.dir}" includes="**/*.html" />
    </yeti>

</target>

Keep in mind that the <yeti> task requires you to have Yeti already installed on the computer that is executing the build script.

PhantomJS

PhantomJS is a headless version of WebKit, the rendering engine that powers Safari and Chrome. As such, it acts very similarly to these browsers (though not exactly the same) and allows you to perform true browser testing without actually opening a browser. PhantomJS comes with scripts to run tests in two different JavaScript testing frameworks: Jasmine and QUnit.

PhantomJS isn’t just a browser—it’s also a scripting environment for that browser. The scripts to run Jasmine and QUnit are part of a suite of scripts that ships with PhantomJS. Both scripts require you to use the appropriate HTML page template for the framework being used.

Installation and Usage

If you’re using Ubuntu, then you can install via apt-get using:

$ sudo add-apt-repository ppa:jerome-etienne/neoip
$ sudo apt-get update
$ sudo apt-get install phantomjs

If you’re using Homebrew on Mac OS X, you can also install PhantomJS via the following command:

brew install phantomjs

For other platforms, download the latest executable for your platform from http://code.google.com/p/phantomjs/downloads/list. Place the entire PhantomJS directory in an easily accessible location (your dependencies directory or someplace else). The scripts to run Jasmine and QUnit tests are in the examples directory.

Note

If you installed using Homebrew or apt-get, then you’ll need to download the files to run Jasmine and QUnit tests from the PhantomJS repository, as these are not included by default.

PhantomJS runs Jasmine and QUnit tests on the command line in the following format:

phantomjs [driver] [HTML file]

For example, to run QUnit:

phantomjs examples/run-qunit.js tests.html

And to run Jasmine:

phantomjs examples/run-jasmine.js tests.html

The results are output onto the command line.

The Ant Target

As with Yeti, the Ant target for PhantomJS is quite simple. There are just a few properties to keep track of:

src.dir = ./src
lib.dir = ./lib
tests.dir = ./tests

phantomjs = phantomjs
phantomjs.driver = ${lib.dir}/phantomjs/examples/run-qunit.js
phantomjs.tests = tests.html

Because the PhantomJS test runners support passing in only one file at a time, the Ant target uses <exec> instead of <apply>:

<target name="test">

    <exec executable="phantomjs" failonerror="true">
        <arg path="${phantomjs.driver}"/>
        <arg path="${tests.dir}/${phantomjs.tests}"/>
    </exec>

</target>

The test target just passes two paths to the executable and fails if there’s an error.

The Buildr <phantomjs> task allows you to perform the same operation with a slightly different syntax:

<target name="test">

    <phantomjs driver="${phantomjs.driver}">
        <fileset dir="${tests.dir}" includes="*.html" />
    </phantomjs>

</target>

The driver attribute is required and specifies the PhantomJS driver to use for your tests. The <phantomjs> task expects one or more <fileset> elements to be present specifying the tests to run. Otherwise, it behaves the same as the previous test target.

JsTestDriver

JsTestDriver is a command-line utility written by engineers at Google. Similar to Selenium and Yeti, JsTestDriver works with already installed browsers to run tests. JsTestDriver has its own JavaScript testing framework as well, so you must write tests using that library by default. There is a QUnit adapter to allow QUnit-based tests to be executed with JsTestDriver, and it’s possible to write your own adapter if you so choose.

Installation and Usage

JsTestDriver is written in Java, so you must first download the latest JAR file and put it into your dependencies directory. JsTestDriver’s most common mode is to run on a developer machine by manually connecting browsers to the JsTestDriver server. Configuration information, including which files to execute as tests, are included in a YAML file that looks like this:

server: http://localhost:4224

load:
    - tests/*.js

The first line indicates where the JsTestDriver server should be set up and the load section indicates which JavaScript files to load for testing.

For a build system, JsTestDriver offers a command that automatically starts and stops browsers all while collecting test results. The format is as follows:

java -jar JsTestDriver.jar --port [port] --browser [browsers] --config [file] 
    --tests all --testOutput [directory]

For example:

java -jar JsTestDriver.jar --port 4224 --browser firefox,iexplore 
    --config conf/conf.yml --tests all --testOutput ./results

This command runs all tests specified in conf/conf.yml on Firefox and Internet Explorer and outputs the results in results. The --browsers option requires the path to the browser executables, so this example assumes that both firefox and iexplore are executable without a full path.

The Ant Target

Creating a target for JsTestDriver requires keeping track of a few key pieces of information:

src.dir = ./src
lib.dir = ./lib
tests.dir = ./tests

jstestdriver = ${lib.dir}/JsTestDriver.jar
jstestdriver.port = 4224
jstestdriver.browser = firefox,iexplore
jstestdriver.config = conf/conf.yml
jstestdriver.output = ./results

The Ant target looks very similar to the one for YUI Test Selenium Driver:

<target name="test">

    <exec executable="java" failonerror="true">
        <arg line="-jar"/>
        <arg path="${jstestdriver}"/>
        <arg line="--port ${jstestdriver.port}"/>
        <arg line="--browser ${jstestdriver.browser}"/>
        <arg line="--conf"/>
        <arg path="${jstestdriver.config}"/>
        <arg line="--tests all"/>
        <arg line="--testOutput"/>
        <arg path="${jstestdriver.output}"/>
    </exec>     

</target>

As currently configured, this Ant target will run all tests specified in the configuration file on Firefox and Internet Explorer.

The Buildr <jstestdriver> task makes it simpler to use JsTestDriver. There are two required attributes: outputdir for the location of the results and config for the location of the configuration file. All other command-line options are present as attributes as well. The following is equivalent to the last example target:

   <target name="test">
    
        <jstestdriver config="${jstestdriver.config}" 
            outputdir="${jstestdriver.output}"
            tests="all" port="${jstestdriver.port}" 
            browser="${jstestdriver.browser}"/>
   
    </target>

Previous Chapter

18. Documentation

Next Chapter

20. Putting It Together