Requesting the Weather

Due to their simplicity, weather displays and stations offer an easy-to-understand illustration of the usage of requests and responses of HTTP with embedded devices. On the hardware side, you could build a simple weather display with Grove components (discussed in Chapter 7). Another option would be to buy a small display and temperature/humidity sensors from eBay, SparkFun, or Adafruit and solder them together.

Gathering the components for a weather display might take some time. In the meantime, you can study its functions by looking at the weather app on your smartphone first. What happens if you request the weather for a certain location?

As you can see on the left side of Figure 9-1, your smartphone acts as a user agent that requests weather data from a public weather database. The smartphone is an “output” device that consists of a display and a network connection.

Figure 9-1. Weather information comes from different locations and has different outputs (displays) and inputs (sensors)

By adding “input” devices with sensors, you could connect your own weather stations to a weather display. This is shown on the right side of Figure 9-1.

Input and output devices do not translate directly to requests and responses of HTTP. According to RFC2616, responses provide a status code and an information body. It is this information body that should be processed and rendered on the display.

JavaScript allows you to prototype a weather display in a web browser, on the command line, and on the embedded devices. Instead of a web browser, the Node.js script acts as “user agent.” The script requests the weather station on behalf of the user. Different weather displays can fetch weather data from the same (public) database as shown in Figure 9-2.

Figure 9-2. Clients for weather data and weather stations

There are different public databases that provide weather data. And at npm you will find different modules to work with weather data. A good option is weather-js.

In a new directory, you first initialize the project and install the module with:

$ npm init -y
$ npm install weather-js

Next, you can set up a script to fetch the weather data from the MSN weather forecast service:

// hello_weather.js
var weather = require('weather-js');
var location = 'Paris, France';
var degreeType = 'C';

weather.find({search: location, degreeType: degreeType}, function(err, result) {
  if (err) {
      console.log(err);
      process.exit(0);
   }
   console.log(result);
});

Behind the scenes, the Node.js modules request and xml2js make an HTTP request and provide the response as a JSON object. Before attaching an embedded device to interact with the weather, let’s quickly check that the API works from the command line.

If you run:

$ node hello_weather.js

You’ll see:

current:
 { temperature: '10',
   skycode: '31',
   skytext: 'Clear',
   date: '2016-03-27',
   observationtime: '22:00:00',
   observationpoint: 'Paris, Paris, France',
   feelslike: '7',
   humidity: '66',
   winddisplay: '28 km/h South',
   day: 'Sunday',
   shortday: 'Sun',
   windspeed: '28 km/h',
   imageUrl: 'http://blob.weather.microsoft.com/static/weather4/en-us/law/31.gif' },
forecast: ...

The next step is to allow users of an embedded device to fetch weather data with the press of a button. Instead of the computer console, we want to display the result to an output (i.e., a screen).

Putting together hardware for a weather display with an Arduino can be as simple as the setup shown in Figure 9-3. You can connect a button to D4 on a Grove shield and an LCD display to the I2C bus.

Figure 9-3. Arduino with push button and LCD display

Programming the screen on a button press is similar to the examples from the previous chapters. With the Johnny-Five library, it looks like this:

$ npm install --save johnny-five

Then, capture a button press as follows:

// weather_display.js
var five = require('johnny-five');
var board = new five.Board();

board.on('ready', function() {

var button = new five.Button(4);

// To display weather information
var lcd = new five.LCD({
   controller: "JHD1313M1"
});

  // Update screen on button press
  button.on('press', function() {
    console.log('press');
    lcd.clear();
    lcd.print("hello");
  });
});

Last, you can add the hello_weather.js script to the Johnny-Five Arduino wrapper:

// weather_display.js
var weather = require('weather-js');
var location = 'Paris, France';
var degreeType = 'C';

var five = require('johnny-five');
var board = new five.Board();

board.on('ready', function() {
    var button = new five.Button(4);
    var lcd = new five.LCD({
       controller: "JHD1313M1"
    });

// on press, fetch the weather
    button.on('press', function() {
        lcd.clear();
        weather.find({search: location, degreeType: degreeType},
                     function(err, result) {
          if (err) {
              console.log("problem fetching weather");
              console.log(err);
              process.exit(0);
           }
        lcd.cursor(0,0).print(location);
        lcd.cursor(1,0).print("temp: "+ result[0].current.temperature);
      });
    });
});

If successful, you’ll see the temperature of the requested place on the LCD.

Prepare the HTTP Client

Before building your own HTTP server that acts as a weather station, it is a good idea to prepare an API client for testing. As shown in Figure 9-4, an API client handles GET requests to fetch state, and POST requests to update state.

Figure 9-4. Different usages for POST and GET requests

On npmjs.com, you’ll find many libraries that help you write custom HTTP clients. The most popular are request and superagent. We’ll use request as it is one of the most popular Node.js modules.

In a new project, install the request module with:

$ npm install --save request

A get request allows you to fetch data via a uniform resource locator (URL).

To have a script act as a user agent:

// get_request.js
var request=require('request');

// take url from command line
var url=process.argv[2];
if (!url) {
  console.log('You must add a path.');
}
console.log('GET url: ' + url);

var headers = {'User-Agent': 'Sensor Agent'};
request.get(url, {headers: headers})
       .on('error', function(err) {
           console.log(err)
   }).pipe(process.stdout);

On Linux systems, you can make the code executable with the hash-bang syntax. In front of the script, you can add:

#!/usr/bin/env node

Now to test this agent, let’s query data from the book website:

$ ./simple_agent.js http://embeddednodejs.com/dummy_data.json
GET url: http://embeddednodejs.com/dummy_data.json
[{"time":"00:00", "data": 0}, {"time":"11:11", "data": 1}]

Similarly, you can invoke POST and PUT requests:

// post_request
var request=require('request');

// the URL for the weather station to come
var url = "http://localhost:4000/measurements";

var headers = {'User-Agent': 'Sensor Agent',
               'content-type': 'application/json'};
var data = {temp: 25};
request({url: url,
         method: 'POST',
         form: data,
         headers: headers})
       .on('error', function(err) {
           console.log(err)
       }).pipe(process.stdout);

You can automate running scripts to fetch or post data with forever:

$ forever start -o log.txt --spinSleepTime 1000 get_request.js

$ npm install -g forever

Adding a Database

To store data from a sensor in the weather station, you need a database. A simple database for embedded devices is SQLite. SQLite is a file-based relational database management system (RDBMS). You can install and run SQLite with embedded Linux.

To connect to the database from JavaScript, you need a connection manager. In Node.js, a good choice is Knex. You can install it as follows:

$ npm install -g knex
$ npm install --save knex
$ npm install --save sqlite3

With the next step, you init a config file for the database connection:

$ knex init
Created ./knexfile.js

The config file can be adapted to your needs. In this case, let’s only use the development version:

// knexfile.js
// update with your config settings.

module.exports = {

  development: {
    client: 'sqlite3',
    connection: {
      filename: './dev.sqlite3'
    }
  }
};

After the database connection is established, Knex provides tools to easily manage tables in a database. For the simple weather station, the plan is to use a database schema with two tables.

One table tracks measurements from a device and another table stores data snapshots from the sensors. In the language of SQL, this means a measurement has many snapshots and a snapshot belongs to a measurement.

To build this schema, Knex can help with simple scripts:

$ knex migrate:make createMeasurements

Then, you add the table data:

// migrations/createMeasurements
var table = function(t) {
    t.increments().primary();
    t.string('name');
    t.string('comment');
    t.timestamps();
}

exports.up = function(knex, Promise) {
  return knex.schema.createTable('measurements', table)
             .then(function () {
                console.log('Measurements table is created!');
             });
};

exports.down = function(knex, Promise) {
  return knex.schema
            .dropTable('measurements', table)
            .then(function () {
               console.log('Measurements table was dropped!');
            });
};

The snapshots table can be created with:

$ knex migrate:make createSnapshots

Then you need to define a schema for the snapshots as follows:

var table = function(table) {
   table.increments().primary();
   table.integer('time');
   table.integer('temp');
   table.integer('measurement_id')
      .references('id')
      .inTable('measurements');
   table.timestamps();
}

exports.up = function(knex, Promise) {
   return knex.schema
              .createTable('weather_events', table)
              .then(function () {
                 console.log('weather events table is created!');
               });
};

exports.down = function(knex, Promise) {
   return knex.schema
               .dropTable('weather_events', table)
               .then(function () {
                  console.log('weather events table was dropped!');
                });
};

To use this connection from other scripts, you need to add “models” with the Bookshelf.js ORM. For a simple weather station, two models will do: one for measurements and one for weather snapshots. Here is the measurements snapshot:

// models/measurement.js

// load the database config
var bookshelf = require('../config');
var Snapshot = require('./snapshot');

var Measurement = bookshelf.Model.extend({
   tableName: 'measurements',
   hasTimestamps: true,

   snapshots: function() {
     return this.hasMany('Snapshot');
   }
});
module.exports = bookshelf.model('Measurement', Measurement);

Next, the weather snapshot:

// models/snapshot.js

// load the database config
var bookshelf = require('../config');
var Measurement = require('./measurement');

var Snapshot = bookshelf.Model.extend({
   tableName: 'snapshots',
   hasTimestamps: true,

   measurements: function() {
     return this.hasMany('Measurement');
   }
});
module.exports = bookshelf.model('Snapshot', Snapshot);

To test this setup, you can create scripts to add new measurements and snapshots to the database:

// scripts/add_measurement.js
var bookshelf = require('./config');
var Measurement = require('./models/measurement');

var measurement = Measurement.forge({name: 'Sensor 1'});
measurement.save().
  then(function() {
    return bookshelf.knex.destroy();
});

Then run:

$ node scripts/add_measurement.js

Now, you can see an entry in the database:

$ sqlite3 dev.sqlite3
SQLite version 3.8.10.2 2015-05-20 18:17:19
Enter ".help" for usage hints.
sqlite> select * from measurements;
1|Sensor 1||1461187585457|1461187585457

And a similar script to add snapshots:

// scripts/add_measurement.js
var bookshelf = require('../config');
var Snapshot = require('../models/snapshot');

var snapshot = Snapshot.forge({measurement_id: 1, temp: 23});
snapshot
  .save()
  .then(function() {
    return bookshelf.knex.destroy();
  });

Last, you need to integrate the database with the web server. Connecting the parts in file server.js can look like this:

// server_with_db.js
var express = require('express');
var bodyParser = require('body-parser');

// connect with db
var bookshelf = require('./config');
var Measurement = require('./models/measurement');

var port = 4000;

var app = express();
app.use(bodyParser.urlencoded({ extended: false }));

// basic routes
app.get('/', function(req, res) {
   res.writeHead(200);
   res.write('server is running');
   res.end();
});

// measurements
app.get('/measurements/:measurement_id', function(req, res) {
   res.writeHead(200);
   Measurement.collection().fetch({
     withRelated: ['snapshots']
   })
   .then(function(collection) {
       return collection.mapThen(function(model) {
         return model.toJSON();
       })
   })
   .then(function(results) {
     res.write(JSON.stringify(results, null, '  '));
     res.end();
   })
   .catch(function(e) {
     console.log(e);
     res.write('problem');
     res.end();
   });
});

app.post('/measurements', function(req, res) {
    var snapshot = Snapshot.forge({measurement_id: req.body.id,
                                  temp: req.body.temp});
    snapshot.save().
      then(function() {
        res.writeHead(200);
        res.write('snapshot saved');
        res.end();
      })
      .catch(function(err) {
        res.writeHead(200);
        res.write('problem saving snapshot');
        res.write(err);
        res.end();
      });
});

app.post('/measurements/:id', function(req, res) {
    var snapshot = Snapshot.forge({measurement_id: req.body.id,
                                  temp: req.body.temp});
    snapshot.save().
      then(function() {
        res.writeHead(200);
        res.write('snapshot saved');
        res.end();
      })
      .catch(function(err) {
        res.writeHead(200);
        res.write('problem saving snapshot');
        res.write(err);
        res.end();
      });
});

var server = app.listen(port);

Now, the whole setup can be tested with the user agents from the weather display. To test this, you run:

// get measurements
$ node get_measurements.js
GET url: http://localhost:4000/measurements/1
[
 {
  "id": 1,
  "name": "Sensor 1",
  "created_at": 1461188045864,
  "updated_at": 1461188045864,
  "snapshots": [
   {
    "id": 1,
    "temp": 23,
    "measurement_id": 1,
    "created_at": 1461188049036,
    "updated_at": 1461188049036
   }
  ]
 }

Now you can post weather data from different places in a network. Let’s take a look at how to post data from embedded devices to the measurement station.

The ws Module

To start, we are going to use the ws module. Install the module with:

$ npm install --save ws

Websockets provide bidirectional communication, which means you can send and receive messages with websockets. This is nice to control device outputs or to receive data from inputs such as sensors.

You can explore the workings of websockets for embedded devices with an Arduino and LED attached. To push data from that device with websockets, write the following:

var five = require('johnny-five');

var http = require('http');
var port = 4000;

var board = new five.Board({
  repl: false
});

// set up WebSocketServer to push bytes
function setupServer(board) {

// import a websocket server
var WebSocketServer = require('ws').Server;

// prepare server
var server = http.createServer(function (req, res) {
  res.write('ok');
  res.end();
}).listen(port);

var wss = new WebSocketServer({server: server});

// connection is set up
wss.on('connection', function(ws) {
    console.log('websocket connected');

  // incoming messages
      ws.on('message', function(message) {
        console.log('received: %s', message);
        var state = parseInt(message);
        state == 1 ? board.led.on() : board.led.off();
      });
  });
}

// start up connection
board.on('ready', function() {
  this.led = new five.Led(3);
  setupServer(this);
});

To test websockets from the command line, you can install a nice command-line tool called wscat:

$ npm install -g wscat

And run:

$ wscat -c ws://localhost:4000
connected (press CTRL+C to quit)
  > 1
  > 0

This should result in switching the LED on the embedded device. Similarly, you can “push” bytes from an embedded device to a websockets client. To see how this works, connect a button to the Arduino on pin 4.

Then you can add a button to the board object as follows:

board.on('ready', function() {
  this.button = new five.Button(9);
  setupServer(this);
});

And, hook into the button press event:

// set up WebSocketServer to push bytes
function setupServer(board) {

var WebSocketServer = require('ws').Server;

// prepare server
var server = http.createServer(function (req, res) {
  res.write('ok');
  res.end();
}).listen(port);

var wss = new WebSocketServer({server: server});

wss.on('connection', function connection(ws) {
    console.log('websocket connected');

      board.button.on('press', function() {
        ws.send('button push');
      });
  });
}

To see how it works, you simply make a connection to the web server. Now you can observe events from the button:

$ wscat -c ws://localhost:4000
connected (press CTRL+C to quit)
  < button push
  < button push
  < button push

Instead of the command-line tool wscat, you can write a small client with the ws module as follows:

var WebSocket = require('ws');

var client = new WebSocket('ws://localhost:4000');

client.on('open', function() {
  client.on('message', function() {
    console.log('push');
  });
});

Websockets are also handy to communicate between an embedded device and a web browser. Connections to the web browser will be explored in the next chapter.

Remote Procedure Calls over Websockets

Based on the WebSocket protocol, you build a medium for remote procedure calls (RPCs). With RPCs, you can invoke methods on objects that come from other devices. Figure 9-6 illustrates how this can be useful.

Figure 9-6. With RPCs, you can invoke functions from remote servers attached to outputs (LEDs, displays) and inputs (sensors)

To begin, take a simple board setup with an LED attached. Without RPCs involved, you can only control the objects within the scope of a script. The initial setup might look like this:

// led_control.js
var port = 4000;

// set up board
var five = require('johnny-five');

// select a board adapter
// var Edison = require('edison-io');
// var Galileo = require('galileo-io');
// var BeagleBone = require('beaglebone-io');

var board = new five.Board({
  // add the adapter here
  // io: new Galileo()
});

board.on("ready", function() {
  var led = new five.Led(3);

  this.repl.inject({
    led: led
  });
});

You can use the LED object on the device with the following commands:

# node led_control.js
1452444373827 Device(s) Intel Galileo Gen 2
1452444373883 Connected Intel Galileo Gen 2
1452444373969 Repl Initialized
>> led.on();

Now, with RPCs you can call the LED object from a different location and context in a network. To see how this works, you must build a web server with support for RPC.

One option is to use the dnode module by James Halliday. You can install this with:

$ npm install --save dnode

Then, wire up the server as follows:

// dnode_server.js
// port to listen to
var port = 4000;

// set up board
var five = require('johnny-five');

// board adapters
// var Edison = require('edison-io');
// var Galileo = require('galileo-io');
// var BeagleBone = require('beaglebone-io');

var board = new five.Board({
  repl: false
});

board.on("ready", function() {

    // place the pin connected to an LED
    this.led = new five.Led(13);

      // now start up server
      startupServer(this);
    });

function startupServer(board) {
  var dnode = require('dnode');
  var net = require('net');

    // create local web server
    var server = net.createServer(function(conn) {

    // set up RPC objects
    var rpc = dnode({

      // these functions can be invoked from other places
      ledOn: function() {
        console.log('on');
        board.led.on();
      },
      ledOff: function() {
        board.led.off();
      }
    });

    // connect local dnode objects with remote
    conn.pipe(rpc).pipe(conn);
    });

      server.listen(port, function() {
        console.log('Server running on port: ' + port);
      });
    }

And the client:

// dnode_client.js
var net = require('net');
var dnode = require('dnode');
var port = 4000;

var rpc = dnode();
rpc.on('remote', function (remote) {
  remote.ledOn(); // this function will be invoked on the server
});

var conn = net.connect(port, 'galileo');
conn.pipe(objects).pipe(conn);

Now you can turn the LED on and off from other places if you run:

$ node dnode_client.js