Table of Contents for

Node.js 8 the Right Way

Procuring External Data

Before we can start manipulating data with Node.js, we have to get it. The data we’ll be using comes from Project Gutenberg, which is dedicated to making public-domain works available as free ebooks.^[34]

Project Gutenberg produces catalog download bundles that contain Resource Description Framework (RDF) files for each of its 53,000-plus books. (RDF is an XML-based format.) The bz2 compressed version of the catalog file is about 40 MB. Fully extracted, it contains a little over 1 GB of RDF files.

To begin, create two sibling directories on your machine, called databases and data.

The databases project directory will hold all of the programs and configuration files you’ll be developing in this chapter. Unless otherwise specified, commands you run will be from a terminal out of this directory.

The data directory will hold the raw data files that we’ll be working with. If you want to put this directory somewhere else for storage reasons, that’s fine, but the examples in this chapter will assume that it’s a sibling of your databases project directory, so modify any paths accordingly.

With that out of the way, open a terminal to your data directory and run the following commands:

This will create a cache directory that contains all the RDF files. Each RDF file is named after its Project Gutenberg ID and contains the metadata about one book. For example, book number 132 is Lionel Giles’s 1910 translation of The Art of War, by Sunzi.

Here’s a very stripped-down excerpt from cache/epub/132/pg132.rdf that shows only the fields that we care about and some surrounding detail:

The important pieces of information that we’d like to extract are as follows:

• The Gutenberg ID (132)
• The book’s title
• The list of authors (agents)
• The list of subjects

Ideally, we’d like to have all of this information formatted as a JSON document suitable for passing in to a document database. For this particular book, our desired JSON would be this:

To get this nice JSON representation, we’ll have to parse the RDF file. On the way there, this provides a great opportunity to explore the BDD pattern.