Throughout the book we have discussed the different ways in which bots can communicate to users. Bots can use text, voice, images, buttons, or carousels. These combined with the right tone and data become a powerful interface for users to quickly and efficiently accomplish their goals. We can easily build text with correct data, but text may not always be the most effective mechanism to communicate certain ideas. Let’s take the example of a stock quote. What kind of data are users looking for when they ask for a quote for, say, Twitter?
Are they looking for the last price? Are they looking for volume? Are they looking for the bid/ask? Maybe they are looking to see what the 52-week high and low prices are. The truth is, each user may be looking for something slightly different. A text description of the stock may make sense to be read by a voice assistant. We would expect Alexa to say, “Twitter, symbol TWTR, is trading at $24.47 with a volume of 8.1 million. The 52-week range is $14.12 to $25.56. The current bid is $24.46, and the current ask is $24.47.” Could you imagine receiving this data in a bot? Parsing through the text is, quite frankly, painful.
An appealing option is to lay out content inside a card, as in Figure 11-1. This sample comes from the TD Ameritrade Messenger bot. A lot of the same data that is included in the text message is communicated via the figure, yet this format is much easier for a human to consume.
Figure 11-1
A stock quote card
A common hero card does not leave much room to create an interface like this. The title, subtitle, and buttons are easy, but the image is not. How do we include such visuals in our bot? In this chapter, we will explore two approaches: custom image rendering using headless browsers and Adaptive Cards, a format that Microsoft’s connectors can render in a channel-specific manner. We will delve into Adaptive Cards first.
Adaptive Cards
When the Bot Framework was first released, Microsoft created hero cards. The hero card, as we explored in Chapters 4 and 5, is a great abstraction over the distinct ways in which different messaging platforms render images with text and buttons. However, it became evident that hero cards are a bit limiting since they are only composed of an image, title, subtitle, and optional buttons.
To provide more flexible user interfaces, Microsoft created Adaptive Cards. The Adaptive Card object model describes a much richer set of user interfaces within a messaging application. It is the channel connector’s responsibility to render an Adaptive Card definition into whatever form is supported by the channel. Basically, it’s a much richer version of the hero card.
Adaptive Cards were announced at the Build 2017 conference. As chat bot developers, we now have one format to describe a rich user interface. The format itself is a mix of a XAML-like layout engine with HTML-like concepts in a JSON format.
Here is an example of a restaurant card and its rendering in Figure 11-2:
In an Adaptive Card, almost everything is a container that can include other containers or UI elements. The result is a UI object tree, just like any other standard UI platform. In this example we have a container with two columns. The first column is double the width of the second column and contains four TextBlock elements. The second column simply contains an image. Lastly, the card includes one action that opens a web URL. Here is another example and its rendering (Figure 11-3):
This has a similar overall layout with two columns that have a 2:1 width ratio. The first column contains text of varying sizes as well as three input fields. The second column contains an image.
We present one more example in Figure 11-4, recalling our stock ticker card discussion.
"speak": "Microsoft stock is trading at $62.30 a share, which is down .32%",
"body": [
{
"type": "Container",
"items": [
{
"type": "TextBlock",
"text": "Microsoft Corp (NASDAQ: MSFT)",
"size": "medium",
"isSubtle": true
},
{
"type": "TextBlock",
"text": "September 19, 4:00 PM EST",
"isSubtle": true
}
]
},
{
"type": "Container",
"spacing": "none",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"width": "stretch",
"items": [
{
"type": "TextBlock",
"text": "75.30",
"size": "extraLarge"
},
{
"type": "TextBlock",
"text": "▼ 0.20 (0.32%)",
"size": "small",
"color": "attention",
"spacing": "none"
}
]
},
{
"type": "Column",
"width": "auto",
"items": [
{
"type": "FactSet",
"facts": [
{
"title": "Open",
"value": "62.24"
},
{
"title": "High",
"value": "62.98"
},
{
"title": "Low",
"value": "62.20"
}
]
}
]
}
]
}
]
}
]
}
Figure 11-4
A stock quote rendering
This template introduces a few more concepts. First, the card has two containers instead of columns. The first container simply displays the two TextBlocks with the data around the company name/ticker and the quote date. The second container contains two columns. One has the last price and change data, and the other has the Open/High/Low data. The latter data is stored in an object of type FactSet, a collection of name-value pairs that are rendered as a tightly spaced group.
The Adaptive Cards website provide a variety of rich samples.1 On the same site, the Visualizer2 makes it clear that Bot Framework chat bots are only a small part of Adaptive Cards. The individual Bot Framework channels are supported with varying degrees of fidelity. The emulator renders the cards faithfully, but many other channels like Facebook Messenger result in images (Figure 11-5).
Figure 11-5
Messenger renders the Adaptive Cards as images
To be fair, Microsoft’s Facebook connector returns a Bad Request (400) status code to any Adaptive Card with unsupported features. This truly captures the dilemma here. Having a common rich card format is a positive development, but only if it is widely supported. Lacking support in a platform like Facebook is detrimental. It is worth noting that the host app allowed in the Visualizer tells a broader adaptive cards story (Figure 11-6).
Figure 11-6
Possible rendering options in the Adaptive Card Visualizer
Note the first seven items (WebChat, Cortana Skills, Windows Timeline, Skype, Outlook Actionable Messages, Microsoft Team, and Windows Notifications) are all systems within Microsoft’s control. Microsoft is building a common format to render cards across its numerous properties.
In short, if your application is targeting many of the Microsoft systems like Windows 10, Teams, and Skype, investing in reusable and consistent cross-platform Adaptive Cards is a good idea.
Microsoft also provides several SDKs to help your custom app render Adaptive Cards. For instance, there is an iOS SDK, a client-side JavaScript SDK, and a Windows SDK; each can take adaptive card JSON and render a native UI from it.
A Working Example
We will now look at a sample to get a better idea of how Adaptive Cards render and how they send input form messages back to the bot. We will use the Emulator as our channel since it implements all the important features. We will use a slightly modified card from a previous example to collect a user’s name, phone number, and e-mail address.
"text": "Don't worry, we'll never share or sell your information.",
"isSubtle": true,
"wrap": true,
"size": "small"
},
{
"type": "TextBlock",
"text": "Your name",
"wrap": true
},
{
"type": "Input.Text",
"id": "name",
"placeholder": "First Last"
},
{
"type": "TextBlock",
"text": "Your email",
"wrap": true
},
{
"type": "Input.Text",
"id": "email",
"placeholder": "youremail@example.com",
"style": "email"
},
{
"type": "TextBlock",
"text": "Phone Number"
},
{
"type": "Input.Text",
"id": "tel",
"placeholder": "xxx.xxx.xxxx",
"style": "tel"
}
],
"actions": [
{
"type": "Action.Submit",
"title": "Submit"
},
{
"type": "Action.ShowCard",
"title": "Terms and Conditions",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"text": "We will not share your data with anyone. Ever.",
"size": "small",
}
]
}
}
]
}
We will also allow the user to click any of two items: a Submit button to send the data and a Terms and Conditions button that displays some extra information when clicked. When a user clicks Submit, the data from the fields is gathered and sent to the bot as an object exposed via the message’s value property. The object sent by the Adaptive Card defined in the previous JSON will have three properties: name, email, and tel. The property names correspond to the field id.
It follows that the code that gets the values is straightforward. It could be as basic as simply checking whether the value exists and executing logic based on it. If we send multiple cards, since they stay in the user’s chat history, it is again critical to ensure a consistent conversational experience.
const bot = new builder.UniversalBot(connector, [
(session) => {
let incoming = session.message;
if (incoming.value) {
// this means we are getting data from an adaptive card
let o = incoming.value;
session.send('Thanks ' + o.name.split(' ')[0] + ". We'll be in touch!");
Figure 11-7 illustrates how this conversation can develop. Note that there is no actual logic within the cards themselves, save for some minor validation. There may be an ability to do so in the future, but for now all such logic must occur in the bot code.
Figure 11-7
An input form Adaptive Card after expanding the Terms and Conditions and clicking Submit
Exercise 11-1
Creating a Custom Adaptive Card
1.
The goal of this exercise is to create a functioning weather update Adaptive Card. You will integrate with a Weather API to provide live weather to your chat bot’s users. Create a bot that collects a user’s location, perhaps simply a ZIP code, and returns a text message echoing the location.
2.
Write the code necessary to integrate with the Yahoo Weather API. You can find information about using it at https://developer.yahoo.com/weather/.
3.
Create an Adaptive Card that includes the various data points that the service provides. The Adaptive Cards website provides two weather samples; you can use one of these if you would prefer. Once done, switch some of the UI elements around in the adaptive card JSON. How easy is it to do so?
4.
Add graphic image elements. For example, display a different graphic to represent sunny versus overcast weather. You may find some assets using an image search online or host some images locally. If you host them locally, make sure you are set up to serve static content.
Well done! You are now able to enrich your bot’s conversational experience with Adaptive Cards.
Rendering Custom Graphics
Adaptive Cards simplify some types of layouts and allow us to declaratively define custom layouts that can be rendered into images. We do not, however, have control over how the image is utilized; as we saw on Messenger, the image is sent as a stand-alone image, devoid of any contextual buttons or text in card format. Among other minor limitations around sizing, margins, and layout control, we do not have a way to generate graphics. Say we wanted to generate a chart to represent a stock price over time. There is no way to do this using Adaptive Cards. What if we had an alternate way of doing this?
The best way to create custom graphics is to utilize technologies we are already familiar with, such as HTML, JavaScript, and CSS! If we could use HTML and CSS directly, we could create custom, branded, beautiful layouts to represent the various concepts in our conversational experience. Using SVG and JavaScript, we would enable us to create stunning data-driven graphics that bring our bot’s content to life.
OK, we are sold. But how do we do this? We’ll take a slight detour into a mechanism we can use to render these artifacts: headless browsers.
A standard run-of-the-mill browser like Firefox or Chrome has many components: the network layer; standards-compliant HTML engines such as Gecko, WebKit, or Chromium; and lastly the UI that allows you to view the actual content. A headless browser is a browser without the UI components. Typically, these browsers are controlled using either the command line or a scripting language. The original and most important use cases that headless browsers address are tasks such as functional tests in an environment where JavaScript and AJAX are enabled. Search engines, for example, can use headless browsers to index dynamic web page content. Phantom3 is an example of a WebKit-based headless browser that was used heavily during the early AngularJS days. Firefox4 and Chrome5 have recently added support for headless modes in both of their browsers. One of the uses that is becoming more common in this space is image rendering. All headless browsers implement a screenshot functionality that we can leverage for image rendering needs.
We will continue with our stock quote example and build something that can return a quote as text. The full working code sample can be found under the chapter11-image-rendering-bot folder in the book’s GitHub repo. To do so, we need access to a financial data provider. One easy-to-use provider is called Intrinio, which provides free accounts to start using their API. Go to http://intrinio.com and click the Start for Free button to create an account to use their APIs. Once we have completed the account creation process, we can access our access keys, which must be passed to the API via Basic HTTP authentication. Using a URL like https://api.intrinio.com/data_point?ticker=AAPL&item=last_price,volume, we receive the last price and volume for AAPL. The resulting data JSON is shown here:
{
"data": [
{
"identifier": "AAPL",
"item": "last_price",
"value": 174.32
},
{
"identifier": "AAPL",
"item": "volume",
"value": 20179172
}
],
"result_count": 2,
"api_call_credits": 2
}
Creating a bot to use this API can be done by using the following code, resulting in the conversation in Figure 11-8:
require('dotenv-extended').load();
const builder = require('botbuilder');
const restify = require('restify');
const request = require('request');
const moment = require('moment');
const _ = require('underscore');
const puppeteer = require('puppeteer');
const vsprintf = require('sprintf').vsprintf;
// declare all of the data points we will be interested in
const datapoints = {
last_price: 'last_price',
last_year_low: '52_week_low',
last_year_high: '52_week_high',
ask_price: 'ask_price',
ask_size: 'ask_size',
bid_price: 'bid_price',
bid_size: 'bid_size',
volume: 'volume',
name: 'name',
change: 'change',
percent_change: 'percent_change',
last_timestamp: 'last_timestamp'
};
const url = "https://api.intrinio.com/data_point?ticker=%s&item=" + _.map(Object.keys(datapoints), p => datapoints[p]).join(',');
const r = _.find(arr, p => p.identifier === ticker && p.item === data_point);
return r;
}
const inMemoryStorage = new builder.MemoryBotStorage();
bot.set('storage', inMemoryStorage);
Figure 11-8
Text stock quotes
Great. We will now create an adaptive card and see how we can utilize what we just covered with headless browsers to render a richer graphic.
For the adaptive card, we will use a template modified from the previous stock update scenario. Instead of sending a string in the endConversation call, we send back a stock card. The renderStockCard function takes the data returned from the API and renders the adaptive card JSON.
Now, if we send a ticker symbol to the bot, we will get a resulting adaptive card. The rendering on the emulator looks good (Figure 11-9). The Messenger rendering is a bit choppy and pixelated (Figure 11-10). We have also uncovered an inconsistency in how the two channels render the “warning” color. We can certainly do better.
Figure 11-9
Emulator rendering of the stock update card
Figure 11-10
Messenger rendering of the stock u-update card
We will now create our own custom HTML template. Now, by trade, as an engineer, I do not do design, but Figure 11-11 is the card that I came up with. We display all the same pieces of data as earlier, but we also add a sparkline for the last 30 days of data.
Figure 11-11
The custom quote card we would like to support
The HTML and CSS for the earlier template is presented here:
Note that we are doing three things that are not obviously possible with adaptive cards: the fine granular control over styling that CSS allows, custom web fonts (in this case, Google’s Roboto font), and an SVG object to draw the sparkline. At this point, all we really must do is modify the appropriate data in the HTML template and render it. How do we do this?
From the different options we mentioned earlier, one of the better options today is Chrome. The easiest way to integrate with headless Chrome is to use the Node.js package called Puppeteer.6 This library can be used for many tasks such as automating Chrome, taking screenshots, gathering timeline data for websites, and running automated test suites. We’ll use the basic API to take a screenshot of a page.
Puppeteer samples use the async/await7 features introduced in Node version 7.6. The syntax waits for a Promise value to return in one line, instead of writing chains of then method calls. The code for rendering an HTML snippet will look as follows:
We launch a new instance of headless chrome, open a new page, set the size of the viewport, load the HTML, and then take a screenshot. The omitBackground option allows us to have transparent backgrounds in the HTML, which result in transparent screenshot backgrounds.
The resulting object is a Node.js buffer. A buffer is simply a collection of binary data, and Node.js provides numerous functions to consume this data. We can call our renderHtml method and convert the buffer into a base64 string. Once we have this, we can simply send the base64 image as part of a Bot Builder attachment.
Constructing the HTML is string manipulation to ensure that the proper values are populated. We add some placeholders into the HTML to make it easy to do string replace calls to place the data into the appropriate locations. A snippet of this is shown here:
The following is the full code to fetch the data from the Intrinio endpoints, read the card template HTML, substitute the right values, render the HTML, and send it as an attachment. Some sample results are illustrated in Figure 11-12.
const msg = new builder.Message(session);msg.addAttachment(card.toAttachment());
session.send(msg);
This renders perfectly fine in the emulator, but we get no result in Messenger. If we look at the Node output, we will quickly notice that Facebook returns an HTTP 400 (BadRequest) response. What’s happening? Although Facebook supports data URIs with an embedded Base64 image, it does not support this format for card images. We can go through the effort of creating an endpoint in our bot that returns the image, but Facebook has yet another limitation: a webhook and the URI for the card image cannot have the same hostname.
The solution is for our bot to host the resulting images elsewhere. A great place to start is a cloud-based Blob store like Amazon’s S3 or Microsoft’s Azure Storage. Since we are focusing on Microsoft’s stack, we’ll go ahead and use Azure’s Blob Storage. We will use the relevant Node.js package.
IMAGE_STORAGE_CONNECTION_STRING is an environment variable that stores the Azure Storage connection string, which can be found in the Azure Portal after creating a storage account resource. After we generate the image into a local file, our code must ensure a blob container exists and create the blob from our image. We then use the new blob’s URL as the source of our image.
renderHtml(data, 584, 304).then(cardData => {
const uniqueId = uuid();
const name = uniqueId + '.png';
const pathToFile = 'images/' + name;
fs.writeFileSync(pathToFile, cardData);
const containerName = 'image-rendering-bot';
blob.createContainerIfNotExists(containerName, {
publicAccessLevel: 'blob'
}, function (error, result, response) {
if (!error) {
blob.createBlockBlobFromLocalFile(containerName, name, pathToFile, function (error, result, response) {
The card is now rendering as expected, as per Figure 11-14.
Figure 11-14
The card now renders!
Exercise 11-2
Rendering Your Graphic Using Headless Chrome
In this exercise, you will take the code from your weather bot from Exercise 11-1 and add custom HTML rendering.
1.
In your adaptive card, add a placeholder that can contain an image to represent the temperature forecast in a chart.
2.
Render an image using headless chrome that shows the forecast using a line chart. You can utilize the same sparkline approach as earlier.
3.
Store the resulting image in blob storage.
4.
Ensure the adaptive card includes the custom rendered image in the designated spot and that it can render in the emulator and Facebook Messenger.
You have now mixed a custom HTML rendering with an adaptive card. No one said we couldn’t do that, right?
Conclusion
In this chapter, we explored two approaches to communicating complex ideas and our chat bot’s brand via rich graphics. Adaptive Cards are a quick way to get started and allow for deeper integration with platforms that support the format natively. Custom HTML-based image rendering allows for much more customization and control over the resulting graphic and is especially valuable where there is no native Adaptive Card support. Both are great choices for highly engaging chat bot experiences.
