The Expectations Game

For more than two years now, a substantial chunk of my conversations with customers has been spent on discussing chat bot capabilities, what they are, and, more importantly, what they are not. Our culture largely confounds chat bot abilities with artificial intelligence, and it is easy to see why. Some chat bots employ rich natural language capabilities, leading us to imagine there is more to them. Likewise, voice-based digital assistants such as Cortana, Alexa, and Google Assistant live in our homes and may be spoken to like real humans. Why wouldn’t chat bots display more intelligence?

The culture is additionally permeated with references to the likes of IBM’s Watson on Jeopardy,⁴ the New York Times’ feature on the Google Brain team⁵ and their feats in language translation using deep learning, self-driving cars, and AlphaZero destroying the world’s highest-rated chess-playing engine after only four hours of learning how to play chess.⁶

These and many other stories highlight the investment and interest in these techniques, foreshadowing the kind of AI-driven interactions with our devices that we are heading toward. Developments in the field of AI have changed the way we interact with, as well as what we expect from, our technology. Assigning human attributes and abilities to our devices is becoming more prevalent. Thinkers in the cognition and science-fiction spaces have long grappled with this possibility as popularized by Asimov’s Three Laws of Robotics, a set of rules that robots obey to ensure the robots don’t go after humans. And now that there are some clear and concrete AI examples in the real world, that kind of reality seems so much closer.

Yet, reality does not match the expectations set forth by AI’s successes in some very specific problem areas. Although we have made tremendous leaps and bounds in terms of natural language processing, computer vision, emotion detection, and so forth, composing all of these pieces into a human-like intelligence, usually referred to as Artificial General Intelligence AGI, is not yet within our grasp and is not a realistic target for chat bots. For every article that celebrates the tremendous achievements in the AI space, there’s a matching article downplaying the hype around the same technology and showing examples of why this type of AI is still far from perfect (think of the articles showing all the images that computer vision algorithms still can’t correctly classify). As with any technology that has been hyped up in the media, we must be reasonable with the expectations we set on it.

Are our bots going to be agents with human-level intelligence having conversations with our users? No. Given the technology and the tasks we want our bots to accomplish, can we make our bots perform those tasks very well? Absolutely. This book aims to equip the reader with the necessary skill to build compelling, engaging, and useful chat bots. It is up to the engineer how much of the latest AI techniques you want to incorporate during this journey. Certainly, these techniques are not required for a great chat bot.

What Is a Chat Bot?

At the most basic level, a chat bot, also referred to simply as a bot throughout this book, is a computer program that can take user input in natural language and return text or rich media to the user. The user communicates with the chat bot via a messaging app, such as Facebook Messenger, Skype, Slack, and others, or via a voice-activated device such as the Amazon Echo, Google Home, or Harmon Kardon’s Invoke powered by Microsoft’s Cortana.

Figure 1-1 illustrates our first bot built using Microsoft Bot Framework. This bot simply returns the same message to the user prefixed by the string “echo: ”. The logic that runs this experience on the Bot Framework is brain-dead simple.

var bot = new builder.UniversalBot(connector, [

    function (session) {

        // for every message, send back the text prepended by echo:

        session.send('echo: ' + session.message.text);

    }

]);

This is a chat bot. Basic and not terribly useful, right? We can just as easily create a YouTube bot that, given user text input, searches for videos on that topic and sends the user links to those videos (Figures 1-2 and 1-3).

This is another basic bot that does just one thing, and it kind of does it well. It integrates with the YouTube API, uses your input as a search parameter, and returns what are referred to in the Bot Framework as cards, something we will explore later in the book. The images make for a richer and more engaging experience—a bit more interesting but still rather basic.

The code for this one is shown next. We make a request to YouTube and translate the response from YouTube format to Bot Framework cards.

OK, how about this? We can have a bot that, given a statement, can tell whether it’s a neutral, positive, or negative statement and return an appropriate response (Figure 1-4). We don’t show it, but the code for this one is as straightforward as the earlier examples: we fetch a sentiment score from a simple sentiment REST API and use it to render an answer.

This is a simple example showing how easy our code can integrate with AI, if we were to go that route. Bots don’t always have to follow a question-response pattern. Bots can reach out to users proactively. For example, we could have a fraud alert bot (Figure 1-5).

A bot can be more task driven. Imagine, for example, a calendar bot that can create appointments, check availability, edit or delete appointments, and give you a summary of your calendar (Figure 1-6).

Now things are starting to get a bit more interesting. We are starting to take natural language and to act on it.

Why Now?

Why are bots becoming such a big deal? Certainly, they have existed in all kinds of incarnations in old-school apps like IRC⁷ and AOL Instant Messenger.⁸ And these were not little experiments. IRC bots have been around for a long time. I remember interacting with quite a few bots over IRC. Being young and naïve when it came to technology, I initially thought there was an actual human responding to my messages. I quickly grasped the idea that there was a machine sitting somewhere responding to what I was writing. The more I interacted with IRC bots, the more I treated them like a command line. This, however, was all pretty niche technology at the time. The public wasn’t interacting with bots on a daily basis so there was no need to cater to natural language interactions.

Today, the way we interact with the technology around us is completely different, and it is driven by three forces: advancements in AI, the idea of messaging apps as a conversational intelligence platform, and voice-activated conversational interfaces.

Why Should We Create Bots?

Why would we want to write bots and use messaging apps as a platform? We could just as easily write mobile apps, publish to the app store, and be done with it, no? Not exactly. There are a variety of trends in user behavior that are making this approach less feasible.

When it comes to some of the bigger brand names, downloading their app is a simple task. I want to use Facebook? Fine, I’ll get the app. I want to check my e-mail; I’ll use the app. But, I want to talk to my local flower shop? I don’t need an app for that. I don’t want an app for that. Why should I download an app for every single business I have contact with? Ideally, I would just be able to call them or, really, just text them, right?

The moves that firms are making in the market are allowing users to talk to a business directly. Let’s take Facebook as an example. A local flower shop can have a Facebook page and enable messaging on the page. Business employees can respond to customer queries in one place. Twitter has a similar feature with its new Direct Message API. That offers a lot of value for businesses. The removal of the app download friction makes it so much easier for users to begin conversing with businesses. The next step, of course, is to automate some of that communication. This is where bots come in. The messaging platform takes care of numerous concerns such as user identity, authentication, overall app stability, and so forth.

This translates to other use cases as well. Let’s take the use case of a productivity tool such as Slack. Slack is a great work collaboration platform that enables people to chat and collaborate with each other across multiple topics. A chat bot on the Slack platform would typically be more productivity oriented. For example, you would probably have a hard time getting people to use a dating bot on Slack as opposed to on a social network like Facebook. Figure 1-7 shows a listing of the top Slack bots. These types of bots are more associated to work tasks such as to-do lists, stand-ups, task assignment, and so on. Clearly, if a team is fully committed and immersed in Slack, creating a bot to carry out common tasks may be more effective than creating an entirely separate web site.

Although Slack’s listing contains a specific category called Bots, the fact is all of these apps are all bots. Some of them might be more conversational, and others could have a more command-line feel to them; as far as we are concerned, a bot is simply listening to messages and acting upon them. For the heavy conversational kinds of chat bots, the topic of natural language understanding, the discipline concerned with understanding human language, is essential for a good user experience. As such, we dedicate the Chapters 2 and 3 to the topic.

Bot Anatomy

Bot Runtime

At the most basic level, a chat bot is a web service that responds to requests from users. Depending on the messaging platform we integrate with, the details differ, but the idea is the same: a messaging platform calls a bot via an HTTP endpoint with a message that contains the user input. Our chat bot’s role is to process the message and respond with a message to the platform that includes the bot’s response plus any attachments or platform-specific data. Figure 1-8 illustrates a generic approach. Depending on the platform, we may be able to return exceptional cases with HTTP status codes or some other format. When our bot processes the message, it responds by calling the channel’s HTTP endpoint. The channel then delivers the message to the user.

There are a few problems with this approach, mainly that we are tying the bot to a specific messaging channel, whereas our bots should be channel agnostic so we can reutilize as much logic as possible. The Bot Framework solves this by providing a connector service that sits between the messaging platform and the bot. In reality, the interaction looks more like Figure 1-9. Note that the channel connector owns the connection and communication with the messaging platform and translates messages into a generic format our bot can recognize. We will cover channels in more detail in the “Channel Integrations” section later in this chapter.

Since the bot runtime is simply a computer program listening to an HTTP endpoint, we can develop the bot using any technology that allows us to receive to HTTP messages. We can use .NET, Node.js, Python, and PHP. In fact, we could simply use the Bot Framework to gain advantage of the connectors and implement the HTTP endpoint using any approach we would like. If we did, however, we would lose out on the Bot Builder SDK. We will cover its benefits and reasons to use it in the “Conversation Engine” section later in this chapter.

Conversation Engine

When building bots, we typically develop a workflow that implements tasks that our bot would like to accomplish. Following the basic thermostat example, we could envision the bot architecture as shown in Figure 1-10.

The workflow always starts with the bot listening for user utterances. An utterance spoken by the user will be resolved to the intents in Table 1-1. If the intent is TurnOn or TurnOff, the bot can execute the right logic and respond with a confirmation message. If we receive a SetTemperature intent, our bot can verify that the Temperature entity exists. If not, we ask the user for it. Once we receive it, we can execute the right logic and send a confirmation response. SetMode would work similarly to SetTemperature in that we would confirm the existence of the entity and elicit it if it does not exist.

This description of what a bot does based on user inputs is a conversation. The activity of designing the types of inputs, the output, and the transitions is called conversational experience design. We cover this topic in depth in Chapter 4.

A conversation engine is the engine that tracks incoming messages, processes them, and executes the state transitions between the conversation diagram nodes (also referred to as dialogs). It does so separately for each user. The state of the conversation is stored so that when the next user message comes into the bot, the bot knows what the user’s current state is. The Bot Framework does a great job of providing the conversation engine via the Bot Builder SDK.

Aside: Intents, Entities, Actions, Slots, Oh My!

There are multiple approaches to developing bots, but they can be summarized into two approaches: bot engine and what I call bot conversation as a service. The bot engine was described earlier: we run our bot as a web service, call into NLU platforms as necessary, and use a conversation engine to route messages to dialogs. The bot conversation as a service approach was popularized by the likes of Dialogflow . The approach implies that the NLU resolution, conversation mapping, state, and transitions occur in the cloud on Dialogflow’s infrastructure. Your bot is then called by Dialogflow to modify responses or integrate with other systems.

When a user’s utterance maps to an intent and a defined set of entities, it is called an action. An action has an intent and a set of parameters. Based on our thermostat bot, we could define an action named SetTemperatureAction. This action is the SetTemperature intent with a Temperature parameter. The type of the Temperature parameter is the Temperature entity. When Dialogflow resolves an action, it can call into your bot to fulfill the action. In this model, the bot logic is focused on the execution of logic based on the NLU service’s resolution logic; the conversation engine is outsourced to the NLU service.

An advanced topic in this type of approach to bot development is slot filling. This is the process through which a service notices that an action was only partially populated by a user input and automatically asks the user to fill in the remaining slot, or what we called action parameters . Tables 1-2 and 1-3 illustrate two sample actions.

Table 1-2

Action Definition for Setting a Temperature in Our Thermostat Bot

Action	Name	Type	Required?	Prompt
SetTemperature	Temperature	Temperature	Yes	What temperature would you like to set?

Table 1-3

A More Complex Action Based on a Flight-Booking Bot

Action	Name	Type	Required?	Prompt
Book Flight	From	City	Yes	Departure city
	To	City	Yes	Destination city
	Date	Datetime	Yes	When are you traveling?

Figure 1-11 illustrates the entire end-to-end flow between the user, messaging platform, connector, NLU service, and the bot in in this conversation as a service model.

The conversation as a service approach can be good at getting something up and running in short order. Unfortunately, this comes at a loss of some control and flexibility. Using the Bot Framework gets around these issues by allowing us full control over the bot engine.

Channel Integration

Building bots means addressing multiple messaging platforms. Your boss asks you to write a Facebook Messenger bot. You release it, and your boss congratulates you for your great work. He then asks you, “Can we add this as a web chat to our FAQ page?” Your bot code is tied to the Messenger Webhooks and the Send API. You waffle around and figure you can isolate some of the logic that communicates to Messenger behind a transport interface. You create a second implementation of the same interface that talks to your chat bot through web sockets. Now you have created your own abstraction of an interface between your bot and a messaging platform.

We want our bot logic to be abstracted away from the individual messaging platforms as much as possible. The details of how to receive messages from the channel and send responses are details we don’t want to concern ourselves with too much, unless we are the professionals building connectors into the various platforms. I don’t think you would be reading this book if you were. You want to develop a bot, not the infrastructure. Lucky for us, the different bot frameworks in the market typically do all of this for us, as illustrated in Figure 1-12. The frameworks allow us to write a bot in a channel-agnostic manner and then connect to those channels by going through a few clicks and entering some data. These features are usually called channels or channel integrations .

As is the case with many generic frameworks, there are some edge cases that the framework does not support because the platform feature is either too new or platform specific. In such cases, the framework should allow us to communicate to the platform in its native format. The Bot Framework provides a mechanism for this.

In addition, our framework should be flexible enough to allow for us to create custom channel connectors. For example, if we desire to build a mobile app that provides a chat bot interface, the framework should allow us to do so. If our enterprise is using an instant messaging channel that is unsupported by Microsoft’s Connector, we should be able to create one. Microsoft’s Bot Framework allows for this level of integration via one of my favorite features: the Directline API.

We will cover channel and custom channel integrations in Chapters 9 and 10.

Conclusion

In this chapter, we took a quick look below the surface of the different components available to build bots. In my work, the Bot Framework has clearly won out against competitors that use a conversation as a service approach. The flexibility and control that the Bot Framework provides is a requirement for many enterprise scenarios. The Bot Framework also provides better and richer abstractions, deeper connector integration, and an open and diverse community. The Bot Framework teams has created an incredibly powerful suite that can be the foundation for any conversational bot. My team and I have been using the Bot Framework for almost two years and have found no reason to abandon the platform. In fact, the framework’s approach to conversational engines and the connector architecture have proven resilient to any use cases we have thrown its way.

For these and many other reasons, this book revolves around using Microsoft’s Bot Framework as the framework of choice. The framework is available for the C#/.NET and Node.js development platforms. For the purpose of this book, we will utilize the Node.js version. We will not utilize any additional tools like TypeScript or CoffeeScript. We simply use vanilla JavaScript to show how easy and straightforward it is to get started writing bots using the Bot Framework SDK for Node.js, aka Bot Builder.

Hype or not, the technology and techniques utilized to build bots are truly amazing. As part of this adventure, I want to make sure that we not only cover the basics of building bots but learn more about some of the underlying techniques and approaches. We will not be diving very deeply into these topics, but I’ll cover enough to give the reader an introductory level understanding of how the intelligence in bots can be implemented to feel comfortable exploring more complex scenarios. In the interest of overall book focus, when I cover such topics, I will provide links and information for additional reading material to complement the content. I am not a data scientist, but I have done my best in introducing the relevant machine learning (ML) concepts.

We are about to embark on an exciting journey though the world of conversational design, natural language understanding, and machine learning as applied to chat bots. As we cover these topics and build bots, keep in mind that these techniques apply to everything from chat bots to voice assistant skills. With natural language and voice interfaces becoming more and more prevalent both at home and in the workplace, I guarantee you will apply these concepts in both current projects and future natural language apps. Let’s get going!

Natural Language Machine Learning Background

The beginnings of the NLP field can be traced back to Alan Turing and, specifically, the Turing test,¹ a test to determine whether a machine can behave intelligently. In the test, an evaluator can ask questions of two participants. Responding as one of the participants is a human; a computer is the second participant. Based on the answers the evaluator receives from the two participants, if the evaluator cannot determine which participant is a human and which one is a computer, then it is said that the computer has passed the Turing test. Some systems have claimed to be able to pass the Turing test, but these announcements have been judged as premature.² There’s a criticism that scripting a bot to try to trick a human to believe that it is human and understanding human input are two very different things. We are quite a few years from getting to passing the Turing test.

One of the most famous early successes in the NLP field was Eliza,³ a simulation of a psychologist written by Joseph Weizenbaum. Written in the mid-1960s, it is a good example of a simple and seemingly intelligent bot. The bot was driven by a script that assigned values to inputs based on keywords and matched the scored input to an output, not dissimilar to recognizers in the Bot Framework. A JavaScript implementation may be found online; see Figure 2-1.⁴ Many other similar systems were created, with varying levels of success.

The NLU engines were generally rule based; they were encoded with structured representations of knowledge for the systems to use when processing user input. Around the 1980s, the field of machine learning started gaining ground. Machine learning is the process of having computers learn without being coded for the task—something seemingly closer to intelligence than the rule-based approach. For example, we briefly explored building a brute-force NLU engine and the tedious work of encoding with the various rules. Using machine learning, our system would not need to know anything about our domain and intent classifications ahead of time, though we can certainly start with a pretrained model. Instead, we would create an engine that we show sample inputs labeled with certain intent names. This is called the training data set . Based on the inputs and labeled intents, we train a model to identify the inputs as the presented labels. Once trained, a model is able to receive inputs it has not yet seen and assign scores to each intent. The more examples we train our model with, the better its performance. This is where the AI comes in: the net effect of training the model with high-quality data is that by using statistical models, the system can start making label predictions of inputs it has not yet encountered.

What was just described is a simplified version of what is known as supervised learning. The name comes from the fact that the input data is labeled. Supervised learning’s performance can be quantitively analyzed quite well because we know the real labels and are able to compare them to the predicted labels to get a quantitative value, a technique known as cross validation . The type of tasks best suited for supervised learning are classification and regression problems. For a class C, classification is the task of determining whether an input i is or is not of class C; for example, is a photo one of a panda bear? Or we can go as far as given a set of classes S, determine the class of input i. Common algorithms for classification include support vector machines and decision trees. Figures 2-2, 2-3 and 2-4 illustrate a typical supervised learning scenario.

Regression is similar but is concerned with predicting continuous values. For example, say we have a data set of some weather features across airports. Maybe we have data for temperature, humidity, cloud cover, wind speed, rain quantity, and the number of flights canceled for that day for JFK in New York, San Francisco International, and O’Hare in Chicago. We could feed this data into a regression model and use it to estimate the number of cancellations given some hypothetical weather in New York, San Francisco, and Chicago.

There are other forms of machine learning besides supervised learning. Unsupervised learning is the task of making sense of unlabeled data, typically data clustering tasks as illustrated in Figures 2-5 and 2-6.

Semisupervised learning is the idea of training a model with some labeled data and some unlabeled data. Reinforcement learning is the idea of a system learning by making observations and, based on said observations, making a decision that maximizes some reward function. If the decision yields a better reward, it is reinforced. Otherwise, the decision is penalized. More information about the different types of learning can be found elsewhere.⁵

There is a fascinating illustration of deep reinforcement learning on Stanford’s CS pages.,⁶ as shown in Figure 2-7. In this demo, an agent navigates a space and learns to navigate toward the red apples with positive reward and avoids the poisoned green apples.

An interesting point to highlight is that the general popular bot applications of NLU and NLP are quite superficial. In fact, there has been criticism to calling what Watson did on Jeopardy or what bots do NLU. As a Wall Street Journal article by Ray Kurzweil stated, Watson doesn’t know it won Jeopardy. Understanding and classifying/extracting information are two different tasks. This is a fair criticism, but a well-built intent and entity model can prove useful when it comes to understanding human language in specific narrow contexts, which is exactly what chat bots do.

Aside from the intent classification problem, NLP concerns itself with tasks such as speech tagging, semantic analysis, translation, named entity recognition, automatic summarization, natural language generation, sentiment analysis, and many others. We will look into translation in Chapter 10 in the context of a multilingual bot.

In the 1980s, interest in artificial neural network (ANN) research was increasing. In the following decades, further research in the area yielded fascinating results. A simplistic view of a neuron in an ANN is to think of it as a simple function with N weights/inputs and one output. An ANN is a set of interconnected neurons. The neural network, as a unit, accepts a set of inputs and produces an output. The process of training a neural network is the process of setting the values of the weights on the neurons. Researchers have focused on analysis of many different types of neural networks. Deep learning is the process of training deep neutral networks, which are ANNs with many hidden layers between the input and output (Figure 2-8).

Google’s Translate, AlphaGo, and Microsoft’s Speech Recognition have all experienced positive results by utilizing deep neural networks. Deep learning’s success is a result of research into the various connectivity architectures within the hidden layers. Some of the more popular architectures are convolutional neural networks (CNNs)⁷ and recurrent neural networks (RNNs).⁸ Applications related to bots may include translation, text summarization, and language generation. There are many other resources for you to explore if you would like to go down the rabbit hole of how ANNs can be applied to natural language tasks.⁹

What is happening within the many ANN layers as the data goes back and forth between the neurons? It seems that no one is quite sure. Google’s Translate, for example, has been observed creating an intermediate representation of language. Facebook’s project to create AI that could negotiate with either other bots or human resulted in the AI creating its own shorthand and even lying. This has been written about as some indication that AI is taking over the world when, in reality, although these are fascinating and discussion-worthy behaviors, they are side effects of the training process. In the future, some of these side effects may become creepier and scarier as the complexity of the networks produce more unintended emergent behavior. For now, we are safe from an AI takeover.

The ease of developing deep learning models by using toolkits such as Microsoft’s Cognitive Toolkit¹⁰ and Google’s Tensor Flow¹¹ is also a significant driver in the recent uptick of popularity of ANN models.

Deep learning techniques are being utilized quite successfully in natural language processing tasks. In particular, speech recognition and translation have benefitted substantially from the introduction of deep learning. In fact, Microsoft Research has created speech recognition software “that recognizes conversations as well as professional human transcribers,”¹² and Google has decreased its translation algorithm’s error rate by between 55 to 85 percent for certain language pairs by taking advantage of deep learning.¹³ However, effectiveness in NLU tasks such as intent classification is not as strong as the deep learning hype may want it to be. The key insight here is that deep learning is another tool in the ML toolkit, not a silver bullet.

Common NLP Tasks

In general, NLP deals with a whole multitude of problems, a subset of which are what we would consider NLU tasks. At a high level, the topics can be related to language syntax, semantics, and discourse analysis. Not every NLP task is immediately relevant to chat bot development; some of them are foundational to the more relevant higher-order features such as intent classification and entity extraction.

Common Bot NLU Tasks

An alternative approach to utilizing preexisting services would be to write your own. This is an advanced topic. If you are reading this book, chances are you do not possess the experience and knowledge to make it work. There are easy-to-use ML packages such as scikit-learn that may give the impression that creating something like this is easy, but the effort requires substantial optimizations, tuning, testing, and operationalization. Getting the right type of performance out of these general-purpose NLU systems takes a lot of time, effort, and expertise. If you are interested in how the technologies work, there are plenty of materials online for you to educate yourself.¹⁴

Cloud-Based NLU Systems

The great news from the space of cloud computing and the investment that the big technology firms are making into the ML as a service space is that the basic functionality for the tasks that we need for our bots are available as services. From a practical perspective, there are many benefits here: developers don’t have to be concerned with selecting the best algorithms for our classification problem, there is no need to scale the implementation, there are existing efficient user interfaces and upgrades, and optimizations are seamless. If you are creating a chat bot and need the basic classification and entity extractions features, using a cloud-based service is the best option.

For our NLU deep dive in the following chapter, we will utilize LUIS. Being a pure NLU system, LUIS has a significant advantage, especially when it comes to Bot Framework integration. Although there are not many benchmarks in the NLU space at this time, LUIS ranks among the top-performing NLU systems in the market.¹⁵

Enterprise Space

There are many other options in the enterprise space—really, too many to list. Some of the bigger company and product names you may run into are IPsoft’s Amelia and Nuance’s Nina. Products in this space are generally advanced and contain years of enterprise-level investment. Some companies focus on IT or other process automation. Some companies focus on internal use cases. Some companies focus on specific verticals. And yet other companies focus entirely on prebuilt NLU models around specific use cases. In some products, we will write bot implementations via a proprietary language versus an open language.

In the end, the decision for enterprises is a classic buy versus build dilemma. Niche solutions may stay around for a while, but it is reasonable to assume that with the amount of investment IBM, Amazon, Microsoft, Google, and Facebook are throwing into this space, companies with less financial backing might be handicapped. Niche players that don’t solve the general bot problem may certainly thrive, and I think we will find more companies creating and innovating in specialized NLU and bot solutions that are powered by the big tech company offerings.

Conclusion

We are truly seeing the democratization of AI in the NLU space. Years ago, bot developers would have to pick up the existing NLU and ML libraries to create a system that could be trained and used as readily and easily as the cloud options we have available these days. Now, it is incredibly easy to create a bot that integrates NLU, sentiment analysis, and coreferences. The firms’ effort behind these systems isn’t something to scoff at either; the largest technology companies are digging into this space to provide the tooling for their users to build conversational experiences for their own platforms. For you as a bot developer, this is great. It means competition will keep pushing for innovation in the space, and as research in the field progresses, improvements in classification, entity extraction and active learning will improve NLU systems’ performance. Bot developers stand to gain from the increased pace of research and improved performance of all these NLP services.

Classifying Intents

We left off within the Build section. In the left pane, we have selected the Intents item. There is one intent in the system: None. This intent is resolved whenever the user’s input does not match any of the other intents. We could use this in our bot to tell the user that they are trying to ask questions outside of the bot’s area of expertise and remind them what the bot is capable of.

A typical workflow for using LUIS is to add an intent and present LUIS with several sample utterances that represent the intent. This is exactly what we will do. Figure 3-3 illustrates the process of creating an intent. The UI allows us to enter the utterance in a free-text entry field. We enter a sample, press Enter, enter another sample, press Enter, and so forth. Once we add enough sample utterances, we click the Save button and we're done with the intent (Figure 3-4).

Note that the user interface allows us to search for utterances, delete utterances, reassign intents to utterances, and display the data in a few different formats. Feel free to explore this functionality as you go along.

Before we add the rest of the intents, let’s see if we can train and test the application so far. Note that the Train button in the top right has a red indicator; this means the app has changes that have not yet been trained. Go ahead and click the Train button. Your request will be sent to the LUIS servers, and your app will be queued for training. You may notice a message that comes up informs you that LUIS is training your app and “0/2 completed.” The 2 is the number of classifier models that your application currently contains. One is for the None intent, and one is for AddCalendarEntry. When training is done, the Train button indicator will turn green to indicate that the app is up-to-date.

The intent interface also gives us information about which intent the latest trained app scores highest for each utterance (Figure 3-5). This piece of data is important because we can easily see when an application is trained to classify an utterance as one intent but assigns the highest score to a different intent. The discrepancy in the trained versus resulting intent is often an indicator that there is something in one or more models that is influencing the result in the wrong direction. We’ll cover this and other scenarios in the Troubleshooting section of this chapter. For now, it appears all our utterances have been successfully trained to result in a score of 1 on the AddCalendarEntry intent and between 0.05 and 0.07 on the None intent (see Figure 3-6); these numbers may vary depending on your exact utterances and also changes made by the LUIS engineering team.

Once trained, we can use the Test slide-out next to the Train button to test the models and see how they respond to different inputs (Figure 3-7). The Batch testing panel link allows a higher volume of testing to be performed. For our purposes, we will stick to the interactive mode.

The way LUIS functions is that it runs each input through all the models that were trained in the Training phase for our app. For each model, we receive a resulting score between 0 to 1 inclusive. The top-scoring intent is displayed prominently. Note that a score does not correspond to a probability. A score is dependent on the algorithm that is being used and usually represents some measure of the distance between the input to an intent’s ideal form. If LUIS scores an input with similar scores for more than one intent, we probably have some additional training to do.

After training our app and testing it, it seems to perform well until we try to break it. Then, it quickly starts looking wrong. Figure 3-8 illustrates this point.

Yikes. This is not terribly surprising. We have trained one intent with a limited number of utterances. We provided zero sample utterances to the None intent. This is the kind of behavior an undertrained model will exhibit. Let’s add some of these silly phrases to the None intent, train, and test again. You may try to add a few more nonsensical test cases like those in Figure 3-9. It should work better. We will not solve for all kinds of issues like this right now. This will take some time, dedication, and user feedback. But we should be aware that training the app what it should not know is as important as training an app what it should know.

Next, we will add the remaining intents. Figure 3-10, Figure 3-11, Figure 3-12, and Figure 3-13 show some sample utterances for the CheckAvailability, EditCalendarEntry, DeleteCalendarEntry, and ShowCalendarSummary intents.

Once all the intents are created and populated with sample utterances, we train and confirm that the predicted intents look accurate. You may note that although the top-scoring intent for each of the utterances is correct, the scores are rather low (Figure 3-14). This is an opportunity for us to train the app further. In fact, we can never assume that we can train an intent to be recognized with such a limited vocabulary and set of data. Getting NLU right requires patience, dedication, and thought. We will add more utterances to our app in an upcoming exercise.

Publishing Your Application

Obviously, we are not yet done developing our app. There are quite a few things missing and many details of LUIS we have not yet explored. We haven’t seen any real user data yet either. But, we can develop both the LUIS app and the consuming application in parallel. The process of taking our trained app and making it accessible via HTTP is referred to as publishing our app.

On the top navigation strip for the app, next to the Build section, we can find the Publish section. When we click this, we are greeted with a page that allows us to deploy the LUIS application (Figure 3-15). LUIS allows us to publish the application in one of two deployment slots: staging or production. Staging is meant for usage when we are still developing and testing the LUIS app. The production slot is meant to be used by production apps. The idea behind the two slots is that you can have a previous stable version of the LUIS app deployed into production, while you work on new app features in the staging slot.

We will go ahead and select the Staging slot from the “Publish to” drop-down. Once it’s published, we can access the app via an HTTP endpoint.

Before we test the resulting endpoint using cURL, a command line tool to transfer data over HTTP (among many other protocols), you may have noticed that below the publish settings there is an Add Key button and a set of keys for several deployment regions. When accessing a LUIS app, we must provide a key, which is how LUIS can bill us for API usage. LUIS is deployed to several regions; a key must be associated with a region. Keys are created using Microsoft’s Azure Portal. Azure is Microsoft’s cloud services umbrella. We will utilize it to register and deploy a bot in Chapter 5. To associate a key with an app, we must use the Add Key button. Lucky for us, LUIS provides a free starter key to use against apps published in the Staging slot.

Once we publish to the Staging slot, a few things happened. We now have information about the app version and the last time it was published. The URL under Starter_Key is now functional. We may enable verbose results (something we will examine momentarily) or Bing spell check integration (which we will discuss later in this chapter) via URL query parameters. Let’s take a closer look at the URL.

The first line of the URL is the service endpoint for the Azure Cognitive Services in the West US region and, specifically, our LUIS app. These are the query parameters that follow:

We can play with the API by making requests and seeing the responses by using curl. At its core, curl is a command-line tool to transfer data over a variety of protocols. We are going to use it to transfer data over HTTPS. You can find more information at https://curl.haxx.se/ . The command we can utilize is as follows. Note that we pass the subscription key as an HTTP header.

This query results in the following JSON. It gives us the score for each intent in our LUIS app.

You may be thinking, whoa, we just learned that we can have up to 500 intents, so the size of this response would be ridiculous. You would be quite correct thinking this (though gzip would certainly help here)! Setting the verbose query parameter to false results in a significantly more compact JSON listing.

Once we are ready to deploy into production, we would publish our LUIS app into the Production slot and remove the staging parameter from the URL request. The easiest way to accomplish this would be to simply have your development and test configuration files point at the Staging slot URL and the production configuration to point at the Production slot URL.

You are of course welcome to utilize any other HTTP tool you are comfortable with. In addition, Microsoft provides an easy-to-use console to test the LUIS API within the API documentation found online.²

Extracting Entities

So far, we have developed a simple intent-based LUIS application. But other than it being able to tell our bot a user’s intent, we can’t really do much with it. It is one thing for LUIS to give us information about the fact that the user wants to add a calendar entry, but it better to be able to tell us for what date and time, where, for how long, and with who. We could develop a bot that asks the user for all these details in a linear sequence whenever it sees an AddCalendarEntry. However, this is tedious and neglects the fact that users may very well present the bot with an utterance like this:

It would be a bad user experience to ask the user to reenter all this data. The bot should immediately know what the datetime value of “tomorrow at 6pm” is and that “Huck” is someone who should be added to the invite.

Let’s start with the basics. How do we make sure that “tomorrow at 6pm.” “a week from now,” and “next month” are machine readable? This is where entity recognition comes in. Lucky for us, LUIS comes equipped with many built-in entities that we can add to our application. By doing so, the datetime extraction will “just work.”

If we go back into the Build section of the LUIS App and click the Entities header, we will encounter an empty list of entities (Figure 3-16). We can add three different types of entities. For now, we will simply add a prebuilt entity. We’ll address normal entities and prebuilt domain entities in later sections.

A prebuilt entity is a pretrained definition that can be recognized in utterances. The entity is automatically tagged in the input, and we cannot change how the prebuilt entities are recognized. There is a good amount of logic in them that we can utilize in our applications, and it is best to understand what Microsoft has built before building our own entities.

There are many different prebuilt entities. Not all entities are available across all supported cultures. The LUIS documentation provides details around which prebuilt entities are available across which cultures³ (Figure 3-17).

Some of these entities include what is called value resolution. Value resolution is the process of taking the text input and converting it into a value that can be interpreted by a computer. For example, “one hundred thousand” should resolve to 100000, and “next May 10th” should resolve to 05/10/2019 and so forth.

You may have noticed the JSON result from LUIS included an empty array called entities. This is the placeholder for all entities recognized in the user’s input. A LUIS app can recognize any number of entities in an input. The format of each entity will be as follows:

The resolution objects may include extra attributes, depending on which entity type was detected. Let’s look at the different prebuilt entity types, what they allow us to do, and what the LUIS API result looks like.

Entity Training

Let’s go back into our application and apply some of what we just learned. Being as we are writing an application related to calendars, the most obvious prebuilt entity of choice for us is datetimeV2. On the Entities page, click “Manage prebuilt entities” and select the datetimeV2, as shown in Figure 3-18.

After adding the entity, we should train our model. In the interactive testing UI, when we enter “add calendar entry tomorrow at 5pm,” we should see the result in Figure 3-19.

That was easy. We publish the application to the Staging slot one more time. Using curl to run the same query, we receive the following JSON:

Perfect. We can now utilize datetime entities in any of our intents. This is going to be relevant for us in all our application’s intents, not just the AddCalendarEntry . In addition, we will go ahead and add the e-mail prebuilt entity, retrain, and publish to the Staging slot again. Now we can try an utterance like “meet with szymon.rozga@gmail.com at 5p tomorrow” to get the kind of result we have come to expect.

Custom Entities

Prebuilt entities can do a lot for our models without any extra training. It would be surprising if everything that we need could be provided by the existing prebuilt entities. In our example of a calendar app, calendar entries, by definition, include a few more attributes that we would be interested in.

For starters, we usually want to give meetings a subject (not only “Meet with Bob”) and a location. Both would be arbitrary strings for meetings subjects and locations. How do we accomplish that?

LUIS gives us the ability to train custom entities to detect such concepts and extract their values from the users’ inputs. This is where the power of the entity extraction algorithms really comes in; we show LUIS samples of when words should be identified as entities and when they should be ignored. The NLP algorithms consider context. For instance, given multiple samples of utterances, we can teach LUIS and ensure it doesn’t confuse Starbucks with Starbuck, the character from Moby Dick.

There are four different types of custom entities that we can utilize in LUIS: simple, composite, hierarchical, and list. Let’s examine each one.

Simple Entities

A simple custom entity is an entity such as a calendar entry subject or the prebuilt e-mail, phone number, and URL entities. One segment of the user input can be identified as an entity of said type based on its position in the utterance and the context of the words around it. LUIS makes it easy to create and train these types of entities. Let’s create the calendar subject entity.

Let’s say we want to be clear when we are telling the calendar bot about a subject name for the entry. Let’s say that we want to accept inputs like “meet with Kim about mortgage application at 5pm.” In this example, the subject will be “mortgage application.” Let’s get this in place.

Navigate to the Entities page and click the “Create new entity” button to create a new simple entity called Subject, as illustrated in Figure 3-20.

Once you click Done, the entry is added to the list of entities in your application. The process of training an entity occurs in the same interface as training intents. Let’s navigate into the AddCalendarEntry intent and add the utterance “meet with Kim about mortgage application at 5pm,” as shown in Figure 3-21. Note that this is just a vanilla utterance and no entities are being identified.

We now mouse over the mortgage and application words and notice that LUIS is allowing us to select the words. Click mortgage and then click application so LUIS has the phrase “mortgage application” selected. The pop-up will list all the custom entity types in your application. Select Subject. The utterance in LUIS should now look like Figure 3-22.

Save the utterance and train your app. At this point, LUIS won’t be that great at identifying subjects quite yet. After all, we just provide one example, and entity identification is more difficult to do properly than intent classification. It needs more samples. We can enter a few more utterances in the utterances editor for the add calendar entry intent. A few samples are shown in Figure 3-23.

Note that no subjects at all were identified. Let’s reinforce the concept. It will take quite a few examples for the system to start recognizing the entity. I added more than ten utterances that had some type of subject somewhere in the utterance, as shown in Figure 3-24. Also, be sure to mark the subject of any utterances you may have added yourself. The process of what I call “bending LUIS to your will” can be more of an art than a science. The key point to remember is that there’s going to be an inflection point at which the algorithms start realizing that something following a word is always an entity until some other key words, based on statistical inference. Think of a scale that you are slowly trying to tip into balance. Our utterances should be carefully crafted to ensure we’re capturing as many variations as possible to show LUIS. Often, each variation will also need to include a few samples to really capture the essence of where within the context of an utterance the algorithm can find specific entities.

After training this data set, we see that the interactive testing tool is getting better at identifying the entity. I entered “hi let’s meet about lawn care and harmonicas at 1:45p” (don’t ask how I came up with that…) and received the result in Figure 3-25. We are making good progress. However, if we start entering inputs of different lengths and variations, LUIS may not identify the entities correctly. It just means we need to further train our entity model. We will leave this as an exercise to the reader.

We now have a good grasp of the calendar subject entity even though there are probably many cases that won’t yet work. And truth be told, you won’t be able to capture all the different types of ways users will ask things until you have a good testing phase. That’s how LUIS app development goes. It is worth looking at the resulting JSON when this application is published.

{

  "query": "hi let's meet about lawn care and harmonicas at 1:45pm",

  "topScoringIntent": {

    "intent": "AddCalendarEntry",

    "score": 0.8653278

  },

  "entities": [

    {

      "entity": "1:45pm",

      "type": "builtin.datetimeV2.time",

      "startIndex": 48,

      "endIndex": 53,

      "resolution": {

        "values": [

          {

            "timex": "T13:45",

            "type": "time",

            "value": "13:45:00"

          }

        ]

      }

    },

    {

      "entity": "lawn care and harmonicas",

      "type": "Subject",

      "startIndex": 20,

      "endIndex": 43,

      "score": 0.587688446

    }

  ]

}

Note that the time entity is being identified as expected. The Subject entity comes back with the relevant entity value. It also comes back with a score. The score in this case is again a similar measure to intent scores; it’s a measure of distance from the ideal entity. Unlike intents, LUIS will not return all your entities and their scores. LUIS will return only simple and hierarchical entities with scores above a threshold. For built-in entities, this score is hidden.

The nice thing about training the entity is that even though the samples with the entity are defined in the AddCalendarEntry intent, they carry over to other intents. Intents and entities are not tied directly to each other. I can say “cancel meeting about olympic hockey” and it works as shown in Figure 3-26.

Another observation is the lower score in terms of identifying the DeleteCalendarEntry intent. We’ve added many more utterances to the AddCalendarEntry intent, but DeleteCalendarEntry and EditCalendarEntry have much fewer examples. Take some time to improve that. Add some alternate phrasings and examples with our new Subject entity before we continue.

Exercise 3-4

Training the Subject Entity and Strengthening Our LUIS App

In this exercise, we will improve our LUIS app by training it to do some additional training.

• Add a Subject entity, as per the directions in the previous section.

• Add utterances into your intents to support the Subject entity. Train and test often to see your progress.

• Aim for at least 25 to 30 samples for LUIS to start. Make sure to convey multiple instances of different ways of expressing ideas.

• Ensure all your intents are getting your attention. Make sure every intent has 15 to 20 samples. Include entities in each intent.

• Train and publish the LUIS app into the Staging slot.

• Use curl to examine the resulting JSON.

Training custom entities, especially ones that are a bit vague in terms of positioning and context, can be challenging, but after some practice, you will start seeing patterns in LUIS’s ability to extract them. Note things that need to be explicitly trained: number of words in the subject, subjects with the word and, subjects followed by datetime, and so forth. You may have noticed the explicit mention of number of samples. These are just starting points. An NLU system like LUIS gets better the more sample data it has. Do not overlook this point. If LUIS is not behaving the way you expect it, chances are it is not a LUIS performance problem but rather that your application needs more training.

The second entity we planned to add was the Location entity . Let’s create a new simple custom entity and call it Location. Like the Subject entity, the location is going to be a free text entity, so we’re going to need to train LUIS with many samples.

We’re going to take a stab at this by adding utterances into the AddCalendarEntry intent again. We need to add utterances in these forms:

Meet with kim to talk about {Subject} at {Location}

Meet about {Subject} at {Location}

Add entry with teddy for {Subject} at {Location}

Add meeting at {Location}

Meet at {Location}

Meet in {Location} at {Subject}

You get it. You should also add datetime instances into these utterances. Training the location is going to be trickier as we are teaching LUIS to distinguish between a location and subject, two concepts that simply need a lot of data for LUIS to begin distinguishing since these are two free-text entities. In the end, I ended up adding more than 30 utterances that contained either just a location or a location combined with other entities. After that amount of training, we get decent performance. I can type “meet for dinner at the diner tomorrow at 8pm” and get the following JSON result:

{

    "query": "meet for dinner at the diner tomorrow at 8pm",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.979418

    },

    "entities": [

        {

            "entity": "tomorrow at 8pm",

            "type": "builtin.datetimeV2.datetime",

            "startIndex": 29,

            "endIndex": 43,

            "resolution": {

                "values": [

                    {

                        "timex": "2018-02-19T20",

                        "type": "datetime",

                        "value": "2018-02-19 20:00:00"

                    }

                ]

            }

        },

        {

            "entity": "the diner tomorrow",

            "type": "Location",

            "startIndex": 19,

            "endIndex": 36,

            "score": 0.392795324

        },

        {

            "entity": "dinner",

            "type": "Subject",

            "startIndex": 9,

            "endIndex": 14,

            "score": 0.5891273

        }

    ]

}

We suggest you take some time to strengthen the entities even further. It would be a good experience to really gain an appreciation for the complexities and ambiguities in natural language and in training an NLU system like LUIS.

Exercise 3-5

Training the Location Entity

In this exercise, you will be adding the Location entity into your LUIS app. You will find that this will take a bit longer than the Subject entity by itself.

• Add a Subject entity, as per the directions in the previous section.

• Add utterances into your AddCalendarEntry to support the Location entity. Train and test often to see your progress.

• Aim to start with 35 to 40 samples for LUIS, probably more. As your intents support more entities, you may have to provide more samples to LUIS to properly distinguish. As you add utterances, constantly train and test to see how LUIS is learning. Make sure to use many variations and examples.

• Publish the LUIS app into the Staging slot.

• Use curl to examine the resulting JSON.

This exercise should have been a good experience in strengthening entity resolution when a single utterance contains many entities.

Composite Entities

Congratulations. The work we have done so far is a significant portion of what LUIS can accomplish. Using the intent classification and simple entity extraction techniques described, we can go off and work on our calendar application. Although we went over simple entities, we quickly ran into some complex NLU scenarios. Without a tool like LUIS, doing this kind of language recognition would be incredibly tedious and challenging.

There is another interesting scenario that comes up in natural language. Our model currently supports a user saying a phrase like this:

"Meet at Starbucks for coffee at 2pm"

What if the user wanted to add multiple calendar entries? What if the user wants to say something like the following utterance?

"Meet at trademark for lunch at noon and at Starbucks for coffee at 2pm"

There’s isn’t anything not allowing a user to say that right now. If we’ve trained our app enough, it will certainly handle this input, and it will identify two Subject instances, two Location instances, and two datetime instances, as shown here:

{

    "query": "meet at culture for coffee at 11am and at the office for a code review at noon",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.996190667

    },

    "entities": [

        {

            "entity": "11am",

            "type": "builtin.datetimeV2.time",

            "startIndex": 30,

            "endIndex": 33,

            "resolution": {

                "values": [

                    {

                        "timex": "T11",

                        "type": "time",

                        "value": "11:00:00"

                    }

                ]

            }

        },

        {

            "entity": "noon",

            "type": "builtin.datetimeV2.time",

            "startIndex": 74,

            "endIndex": 77,

            "resolution": {

                "values": [

                    {

                        "timex": "T12",

                        "type": "time",

                        "value": "12:00:00"

                    }

                ]

            }

        },

        {

            "entity": "culture",

            "type": "Location",

            "startIndex": 8,

            "endIndex": 14,

            "score": 0.770069957

        },

        {

            "entity": "the office",

            "type": "Location",

            "startIndex": 42,

            "endIndex": 51,

            "score": 0.9432623

        },

        {

            "entity": "coffee",

            "type": "Subject",

            "startIndex": 20,

            "endIndex": 25,

            "score": 0.9667959

        },

        {

            "entity": "a code review",

            "type": "Subject",

            "startIndex": 57,

            "endIndex": 69,

            "score": 0.9293087

        }

    ]

}

And yet, parsing this using code would be quite challenging. How do we tell which entities should be grouped together? Which location goes with which subject? You should be able to use the startIndex property to figure it out I suppose, but that’s not always as obvious.

Lucky for us, LUIS can group the entities into what are called composite entities . Rather than the messy result shown previously, LUIS will tell us which entities are part of which composite entity. This makes it way easier for us to know that there were two separate AddCalendar requests, one for 11 a.m. coffee at Culture and another one for a code review in the office at noon.

Composite entities can be created on the Entities page of LUIS. Figure 3-27 illustrates the process. Click the Create new entity button, enter a name for the entity, select the Composite entity type, and select the child entity types to be included as part of the new entity. We will use the name CalendarEntry to identify our composite entity.

Once it is created, we need to properly train LUIS to recognize it. Let’s look at the AddCalendarEntry intent again. The easiest way to train LUIS would be to find all utterances that have the required three entities and wrap the entities into the composite entity. Figure 3-28 shows an example.

Click the first Location entity . A pop-up will appear asking you to relabel the entity or wrap it in a composite entity. Click “Wrap in composite entity” (Figure 3-29).

We move our mouse over the Subject and datetimeV2 entities . Note the green underline expands to cover each entity (Figure 3-30). Click datetimeV2 so that it is included in the composite entity and click the CalendarEntry name.

Do the same for the second instance of the CalendarEntry entity . The result should look like Figure 3-31.

We should do the same for any other utterance we can find that includes the three entities. Once we train and publish the app, LUIS should start extracting this composite entity. We only show the relevant API section here:

"compositeEntities": [

    {

        "parentType": "CalendarEntry",

        "value": "culture for coffee at 11am",

        "children": [

            {

                "type": "builtin.datetimeV2.time",

                "value": "11am"

            },

            {

                "type": "Subject",

                "value": "coffee"

            },

            {

                "type": "Location",

                "value": "culture"

            }

        ]

    },

    {

        "parentType": "CalendarEntry",

        "value": "the office for a code review at noon",

        "children": [

            {

                "type": "builtin.datetimeV2.time",

                "value": "noon"

            },

            {

                "type": "Subject",

                "value": "a code review"

            },

            {

                "type": "Location",

                "value": "the office"

            }

        ]

    }

]

Exercise 3-6

Composite Entities

In this exercise, you will add composite entities to your LUIS app.

• Create a composite entity called CalendarEntry, composed of datetimeV2, Subject, and Location entities.

• Train every utterance that has these three entities to recognize the composite entity.

• Train additional examples with multiple instances of the CalendarEntry composite entity . Remember, it takes time, dedication, and persistence to get it right.

• Publish the LUIS app into the Staging slot.

• Use curl to examine the resulting JSON.

Composite entities are a great feature to group entities into logical data objects. Composite entities allow us to encapsulate more complex expressions.

Hierarchical Entities

A hierarchical entity allows us to define a category of entities and its children. You can think of hierarchical entities as defining a parent/subtype relationship between entities. We have run into this type before. Do you recall the Datetimev2 entity? It had seven subtypes such as daterange, set, and time.

LUIS allows us to easily create our own subtypes. Say we wanted to add support in our model to specify the calendar entry visibility as public or private. We could add support for utterances like this:

"create private entry for interview with competitor at starbucks"

 "create invisible entry for interview with recruiter at trademark"

The words private or invisible here indicate the visibility field of the calendar. Why would we create a hierarchical entity as opposed to a simple entity? Can’t we just look at the value of a Visibility property and determine whether it should be a private meeting or not? Yes and no. If the user sticks to those two words, yes. But remember, natural language is ambiguous and vague. Phrasings change. The user can say invisible, private, privately, hidden. It’s the same with public. If we make assumptions about a closed set of options in our code, then we would have to change our code any time a new option shows up. The reason a hierarchical entity should be used as opposed to a simple one is that the statistical models of where in context the hierarchical entity appears is shared by the subtypes. Once that is identified, the step of identifying the child entity is essentially a classification problem. Making the entity hierarchical makes for better LUIS performance versus two simple entities. Not to mention, it’s more efficient to have LUIS classify the meaning of an entity in the context of our application rather than writing code to do so.

Figure 3-32 illustrates the creation of a new hierarchical entity. We do this by visiting the Entities page, clicking “Create new entity,” and selecting Hierarchical from the entity type drop-down. We give the parent entity a name and add the child entities. Once we click Done, it is a matter of going into the intent utterances and training LUIS. Let's go into AddCalendarEntry and add a few samples.

You may notice that one or two samples are not sufficient. We need to give LUIS a really good idea of where and how it may encounter the public and private visibility modifiers before it can start recognizing the entity in our inputs. The ten samples in Figure 3-33 were a good start.

Once we train and publish , we can view the resulting JSON via curl, as shown here:

{

    "query": "create private meeting for tomorrow 6pm with teddy",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.9856489

    },

    "entities": [

        {

            "entity": "tomorrow 6pm",

            "type": "builtin.datetimeV2.datetime",

            "startIndex": 27,

            "endIndex": 38,

            "resolution": {

                "values": [

                    {

                        "timex": "2018-02-19T18",

                        "type": "datetime",

                        "value": "2018-02-19 18:00:00"

                    }

                ]

            }

        },

        {

            "entity": "private",

            "type": "Visibility::Private",

            "startIndex": 7,

            "endIndex": 13,

            "score": 0.9018322

        }

    ]

}

{

    "query": "create public meeting with jeff",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.975892961

    },

    "entities": [

        {

            "entity": "public",

            "type": "Visibility::Public",

            "startIndex": 7,

            "endIndex": 12,

            "score": 0.6018059

        }

    ]

}

List Entities

So far, the prebuilt, simple, composite, and hierarchical entities were all extracted from user input via machine learning techniques . Every time we added one of these entities and trained LUIS , you may have noticed the number of models being trained increased. Recall that a LUIS application is composed of one model per intent/entity. By now, we should be at ten models. Each of these is rebuilt any time we train our app.

List entities exist outside this machine learning world. A list entity is simply a collection of terms and synonyms for those terms. For example, if we want to identify cities, we can add an entry for New York that has the synonyms NY, The Big Apple, The City That Never Sleeps, Gotham, New Amsterdam, etc. LUIS will resolve any of these alternate names into New York.

Once a custom list entity type is created, we are redirected to a list entity editor in which we can enter the canonical term and the synonyms. This interface allows us to add new terms and their synonyms. It also makes recommendations to add extra terms that seem related to what we have added thus far. List entities are limited to 20,000 terms, including synonyms. We can have up to 50 list entities per application, so there is a lot of potential for LUIS-based term and synonym lookup features. Figure 3-34 shows a sample custom list entity definition.

Since list entities are not learned by LUIS, new values are not recognized based on context. If LUIS sees “Gotham,” it identifies it as New York. If it sees “Gohtam,” it does not. It is literally a lookup list.

{

    "query": "meet in the big apple",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.943692744

    },

    "entities": [

        {

            "entity": "the big apple",

            "type": "Cities",

            "startIndex": 8,

            "endIndex": 20,

            "resolution": {

                "values": [

                    "New York"

                ]

            }

        }

    ]

}

When using the API, LUIS will highlight the term that matches a list entity type and will return the canonical name in the resolution values. This allows your consuming application to ignore all the possible synonyms for a term and execute logic based on the canonical names. List entities are powerful for situations where you know the set of possible values for terms ahead of time.

Prebuilt Domains

I hope you have gained an appreciation for some of the challenges of building NLU models . Machine learning tools allow us to get computers to start learning, but we need to be sure we are training them with a lot of good data. It takes years of day-to-day interactions for humans to be immersed in a language to be able to truly understand it. Yet, we assume that AI means that a computer will be able to pick up the concepts with ten samples. When it doesn’t, sometimes we think to ourselves, “Oh, come on, you should know this by now!”

To help us on our journey, many of the NLU platforms provide what are called prebuilt models or domains . Essentially, the creators of LUIS and other platforms want to give us a head start with some domains that we can easily include in our application, train LUIS, and be off to the races. Some of LUIS’s prebuilt models are shown in Figure 3-35.

We can find prebuilt domains in LUIS by navigating into the Build section and clicking the Prebuilt Domains link in the bottom left. At the time of this writing, this feature is still in Preview mode. That is the reason it is so isolated and why it is dynamic and may change by the time you read this. LUIS includes a variety of domains from Camera to Home Automation to Gaming to Music and even Calendar, which is similar to the app we have been working on in this chapter. In fact, we will do just that in Exercise 3-7. The “Learn more” text links to a page that describes in detail what intents and entities each domain pulls in and which domains are supported by which cultures.⁴

When we add a domain to your application, LUIS will add all the domain’s intents and entities into our application, and they will count toward the application’s maximums. At that point, we able to modify the intents and entities as we see fit. Sometimes you may want to get rid of certain intents or add new ones to complement the prebuilt ones. Other times we may need to train the system with more samples. We suggest the prebuilt domains are treated as starting points. Our goal is to extend them and build great experiences on top of them.

Phrase Lists

So far, we have been exploring different techniques to create great models. We have the tools we need to make sure we can create a good conversational experience for our users. There are cases when we train LUIS that the model performance is not as good as we would like. Entities may not be getting recognized as well as we would like them to. Maybe we are building a LUIS app that deals specifically with internal terms that aren’t exactly part of the culture your application is using. Maybe we haven’t had a chance to train LUIS entities with every known possible value for an entity and list entities don’t cut it because we want our entities to remain flexible.

One way to improve LUIS performance under these circumstances is to use phrase lists. Phrase lists are hints, rather than strict rules, that LUIS uses when training our app. They are not a silver bullet but can be very effective. A phrase list allows us to present to LUIS a category of words or phrases that are related to each other. This grouping is a hint to LUIS to treat the words in the category in a similar way. In the case of an entity value not being recognized properly, we could enter all the known possible values as a phrase list and mark the list as exchangeable, which indicates to LUIS that in the context of an entity, these values can be treated in the same way. If we are trying to improve LUIS’s vocabulary with words it may not be familiar with, the phrase list would not be marked as nonexchangeable.

Let’s say we wanted to improve our Calendar model’s private visibility entity performance. After all, there are many ways of expressing that we want a private meeting. As a starting point, we could add a phrase list with all the different words we could expect the model to see. Figure 3-36 shows the LUIS user interface for working on a phrase list. You can get here by selecting the Phrase Lists item under the Build page and clicking Create new phrase list.

A phrase list requires a name and some values. We enter the values one by one in the Value field. As we press Enter, it adds them to the Phrase list values field. The Related Values field contains synonyms automatically loaded by LUIS. We then select the checkbox to tell LUIS that the values are interchangeable.

Before training, let’s try a few variations of the private meeting utterances without the phrase list enabled. If you try utterances like “Meet in private,” “Meet in secret,” or “Create a hidden meeting,” LUIS does not recognize the entity. However, if we train the app with the phrase list, LUIS has no problem identifying the entity in those samples and many others.⁵

If you have completed this exercise successfully, congratulations! You are getting darn good at using LUIS.

Active Learning

We’ve spent weeks training a model , we’ve gone through a round of testing, we’ve deployed the application into production, and we’ve switched our bot on. Now what? How do we know if the model is performing the best it can? How do we know whether some user has thrown unexpected input at the our application that breaks our bot and results in a bad user experience? Bug reports are one way for sure, but we would depend on getting that feedback. What if we could find out about these problems as soon as they occur? We can do so by taking advantage of LUIS’s active learning abilities.

Recall that supervised learning is machine learning from labeled data, and unsupervised learning is machine learning from unlabeled data. Semisupervised learning lives somewhere in between. Active learning is a type of semisupervised learning in which the learner asks the supervisor to label new data samples. Based on the inputs that LUIS is seeing, it can ask you, the LUIS app trainer, for your assistance labeling data that is coming from your users. This improves model performance and over time makes our application more intelligent by using real user input as sample data.

You can access this functionality through the Review endpoint utterances link on the Build page (Figure 3-37). Throughout the training of the application, we’ve been utilizing the published application endpoint to test various utterances. LUIS bases its active leaning on the inputs against the endpoint, not the Interactive Test feature.

The interface allows us to review past utterances and their top-scoring intent, referred to as the aligned intent . As trainers, we can add the utterance to the alignment intent, reassign to a different intent, or altogether get rid of the utterance. We can also zero in on specific intents or entities if we know there are problems with any of them.

Before adding the utterance to the aligned intent, we need to confirm that the utterance is correctly labeled and any entities are being correctly identified. We suggest that using this interface to improve LUIS application is a common practice for any team .

Dashboard Overview

Now that we have trained our application and utilized it for testing, it is well worth highlighting the data that the dashboard provides. The dashboard allows us to get a good glance at the overall app status, its usage, and the amount of data we have trained it with.

The very top provides information about the last time we trained and published the application, as per Figure 3-38. We can also get some metrics about the number of intents and entities we are using, the number of list entities we have, and how many total labeled utterances our application has so far.

The next section displays the kind of usage that the application is getting through the API. We can monitor the amount of endpoint hits for the last week up to the last year. This data is available only once an application is published to the production slot. This is illustrated in Figure 3-39.

Lastly, we are presented with an intent and entity breakdown, as shown in Figure 3-40. Here we see a distribution of the percentage of utterances used to train each intent. You can clearly see some of our intents contain significantly more sample utterances than others. It’s the same for entities. The uneven distribution does not necessarily mean that an entity or intent needs more training.

Managing and Versioning Your Application

Everything we have done so far is part of the common workflow of adding samples, training, and publishing a LUIS application. During the development phase, this workflow is repeated over and over again. Once your application is in production, you should be careful about what you do to your app. The process of adding a new intent or entity can have unforeseen effects on the rest of the application, and it is best that editing an existing application is done in isolation so it can be tested properly.

We have experience with the concepts of the staging and production deployment slots. This certainly helps; we know that we can test changes without publishing to our production endpoints. A common rule is to have the Staging slot host the dev/test version of the application and the Production slot host the production version. Whenever a new application is ready for production, we move it from the Staging slot to the Production slot. But what if we make a mistake in our models? What if we need to roll the Production slot back? That is where versions come in.

LUIS allows you to create a named version of the application at any point in time. So far, by default we have been working on version 0.1. Once it is ready for production, we can publish it and clone it into a new version 0.2. At that point, you set the 0.2 version to Active. Now, the LUIS interface is editing version 0.2. If we accidentally publish version 0.2 into the production slot, we can easily go back to version 0.1 and publish that. Once version 0.2 is production ready, we deploy that into the Production slot and clone it into version 0.3 and set that version as the active version. And so forth. If at any point you deploy a version into the Production slot and need to revert, you set your LUIS active back to 0.2 and publish that version into the Production slot. The workflow is illustrated in Figure 3-41.

We access the application version information through the Settings page. Figure 3-42 and Figure 3-43 show the interface plus what it looks like after cloning version 0.1 into 0.2.

Note that after closing 0.1, it remains in the Staging slot, but 0.2 becomes the Active version. LUIS also doesn’t allow for easy branching. If multiple users want to make changes to a single version, they cannot create a new version and then merge their changes using the user interface. One way to accomplish this would be to download the LUIS App JSON by clicking the Export Version button in Figure 3-42, utilizing a source control tool like Git to branch and merge, and finally, using the “Import new version” button to upload a new version from a JSON file.

The same page also allows us to add collaborators to the application. This is a great way to give access to other folks in your organization to assist in editing, training, and testing versions of the app. At the time of this writing, there are no fine-tuned audit controls; all collaborators can do anything with the application except add/remove other collaborators (Figure 3-44).

Integrating with Spell Checking

One advanced feature in LUIS is the ability to integrate with a spell checker to automatically fix misspellings in user input. User input is, by its very nature, messy. Misspellings are immensely common. Combine that with the common usage of messaging apps, and you have a recipe for consistent misspelled input.

The spell checker integration runs the user query through Bing’s Spell Checker service, gets a possibly altered query with misspellings fixed, and runs that altered query through LUIS. This feature is invoked by including the query parameters spellCheck and bing-spell-check-subscription-key. You can get a subscription key from the Azure Portal, which we will introduce in Chapter 5. We will also utilize the Spell Check API more directly in Chapter 10.

This functionality can be helpful, and we would typically recommend it with a caveat. If our entities contain domain-specific values or product names that are not strictly part of the English language, we may get an altered query in which LUIS is unable to detect an entity. For example, it may break up one word into multiple words when such behavior is unwanted. Or, if our application is expecting financial tickers, it may just change them. For example, VEA, a Vanguard ETF , is changed to VA. In the United States, that’s a common reference to the state of Virginia. The loss of meaning is quite significant; I advise caution in using this feature.

The effect of the spell check on the LUIS API result is easy to spot. The result now includes a field called alteredQuery. This is the text passed into the LUIS models. A sample curl request and response JSON is presented here:

Import/Export Application

Any application built in LUIS can be exported into a JSON file and imported back into LUIS . The JSON file format is exactly what we would expect. There are elements that define which custom intents, custom entities, and prebuilt entities the application uses. There are additional elements to capture phrase lists. And, not surprisingly, there is a rather large segment describing all the sample utterances, their intent label, and the start and end index of any entities in the utterance. We can export the application by clicking Export App in the My Apps section of LUIS or Export Version in the Settings page, as per Figure 3-41.

Although the format of the exported application is specific to LUIS, it is easy to imagine how we could write code to interpret the data by other applications. From a governance perspective, it is good practice to export our applications and store the JSON in source control because the action of publishing an action is irreversible. This should not be an issue if our teams follow a strategy in which a publish into the Production slot implies the creation of a new application version, but mistakes do happen.

One of the most common questions we receive in our work with LUIS is “why we can’t import an application into an existing application?” The reason is that this would be tantamount to a smart merge, especially where there are overlapping utterances with different intents or same name intents with completely different application connotations. Since every application has different semantics, this merge would be a nontrivial task. We suggest either utilizing Git to manage and merge application JSON code or creating custom code to merge using the LUIS Authoring API.

Using the LUIS Authoring API

The API is very rich and allows for training, custom active learning, and enables CI/CD type scenarios. The API Reference Docs⁶ are a great place to learn about the API.

Troubleshooting Your Models

Building LUIS applications can be more of an art than science. You will sometimes spend a lot of time teaching LUIS the difference between some entities or where in a sentence an entity can appear. Be patient. Be thorough. And always think of the problem in statistical terms; the system needs to see enough samples to truly start understanding what’s happening. As people, we can take our intelligence and language understanding for granted. In relative terms, it is quite amazing how quickly we can train a system like LUIS. Remember this as you work with LUIS or any other NLU system.

Conclusion

That was quite a lot of information! Congratulations, we are now equipped to start building our own NLU models using a tool like LUIS. To recap, we went through the exercise of creating an application by utilizing prebuilt entities, custom intents, and custom entities. We explored the power of the various prebuilt entities and dabbled a bit in the prebuilt domains that LUIS provides. We spent time training and testing our application, before publishing it into different types of slots and testing the API endpoints using curl. We optimized our application using phrase features and further improved it by using LUIS’s active learning abilities. We explored versioning, collaborating, integrated spell check, exporting and importing of applications, using the authoring API, and common troubleshooting techniques in our LUIS applications.

I must reiterate that the concepts and techniques you just learned are all applicable to other NLU platforms. The process of training intents and entities and optimizing models is a powerful skill to have in your toolkit, whether for bots, voice assistants, or any other natural language interface. We are now ready to start thinking about how we build a bot. As we do, we’ll keep checking back into this LUIS application as it gets consumed by our bot.

Common Use Cases

Developers are creating all sorts of conversational experiences. We can experience bots that specialize in tasks such as selling items, answering questions about products, sending order statuses, answering inquiries about orders, provisioning cloud infrastructure, searching over multiple data sources, sharing cat GIFs, and doing millions of other things.

At a high level, we will split the bots into two larger categories: consumer and enterprise. There is of course overlap in the subcategories but also some clear dividing lines.

Common Consumer Cases

Consumer bots are typically available via channels such as Facebook Messenger, Slack, and the other public messaging apps; web chat; voice interfaces; or even custom mobile apps when a custom interface is required. On the lower end of the quality scale, they are no more than toys. On the higher end, they can be impressive feats of design and engineering. Because of the general AI and bot fever we discussed in Chapter 1, many companies are deploying a bot along with their products. Atlassian, for example, has a Slackbot for its JIRA product. Even Amazon has a chat bot integrated into its mobile shopping app. You will also find brands dipping their toes into bots via Facebook Messenger. Facebook Pages makes it easy for a company to have an outward-facing public channel to talk to its customers via either public posts or Messenger. If it is Messenger, a human agent needs to log into the page inbox and reply to each message. A first step for many companies is to deploy a Messenger bot that replies to a few types of user queries, with the rest simply left for humans to reply to. Utility-wise, we are still trying to answer the question, what makes the most sense for users? The variety of bots in the space certainly points to that. The following are some broad categories of effective approaches.

FAQ Bot

An FAQ bot is typically the first entry into the bot and NLP space by teams taking the technology for a test run. It is an easy use case: let’s take our existing FAQ and place it as a bot on Facebook Messenger or enterprise messaging. That way, the most typically asked questions can be caught by a bot before an employee spends time answering them. A simple text-based FAQ bot can turn into something quite interesting and aesthetically pleasing from a user perspective. An answer to a commonly asked question doesn’t simply have to be a block of boring text. The answer can include further content such as images, videos, and links to additional content.

For example, consider a financial services bot that can answer different types of questions about financial topics. Within its response, it can embed additional suggested topics of interest as buttons. At that point, the user can look at related terms and their definitions. If there are websites that visually represent a concept, for example, the iron condor option investment strategy, those links can be included in the response for the user to click to get more information. Of course, our conversation design needs to balance all that content with possible user overload. The sweet spot in between can be effective at providing the user with a pleasant experience with the bot. Figure 4-1 is an example of Child Fund International’s FAQ bot embedded into a web page.

Task-Oriented Bot

A task-oriented bot is a virtual agent that can help users with a variety of tasks specific to a domain. These types of bots are sometimes called concierge bots . For example, JIRA’s Slackbot (Figure 4-2) is task oriented. It can create tasks and assign tasks based on a conversation a team is having.

I once worked on a diabetes coach chat bot, which could help users who have Type 2 diabetes ask for meal and exercise advice personalized according to previous conversations and other data about the users. There are also financial services bots that connect to a trading account and update the user on their account balances and positions and even trade, like the TD Ameritrade bot (Figure 4-3). The Calendar LUIS app we developed in Chapter 3 is the base for a calendar task bot.

Broadcast Bot

A broadcast bot is an interesting concept and is quite common. We can think of this as a bot that reaches out to the user without prompting, as opposed to the user contacting the bot first. In some bots, it is more a pattern to keep bots engaged. For example, different news bots, like the CNN bot on Facebook Messenger, will reach out daily with the biggest stories of the day.

A subset and more nuanced version of this can be seen in some celebrity bot implementations. Typically, these types of bots exist for fun. They adopt the personality of a celebrity and can talk to users about topics of interest, products, and other ways of interacting with the celebrity’s branch. The bot can navigate you through a script of topics, send you videos and images, and maybe talk about products that the celebrity is endorsing. The conversation is almost entirely driven by the bot, instead of the user. It is an interesting storytelling device , but its success comes down to consistent fresh content. Figure 4-4 shows an example of Project Cali, a Snoop Dogg bot created for fun.

E-commerce Bot

Although not yet big in North America, bots are slowly starting to sell products to consumers. It is not a terribly challenging task from a technical perspective; the bigger challenge is getting users to use a messaging instead of apps or websites. The amount of e-commerce integration in these kinds of bots varies. For example, some bots provide the complete end-to-end shopping experience. Looking at clothing items (Figure 4-5) or flowers (Figure 4-6) through a bot is different from an online shopping experience. Some bots lean into this and provide quirky or innovative ways of figuring out what products to show the user to get the impulse buy!

We also run into experiences where the bot is responsible only for broadcasting a receipt for a purchase and order status updates, with a limited set of bot functionality. Everything else gets automatically routed to a human customer support representative. Although this kind of experience is not fully integrated e-commerce, it is a great first step into that journey and into getting customers acquainted with bots. In short, companies are embracing what is being called the digitally driven consumer journey , and bots are part of this strategy.¹

Different messaging platforms provide different levels of payment support. We could certainly create e-commerce via a bot by providing a custom checkout page where the user can enter their payment information. The conversation is paused at this point. Once the payment is processed, a message is sent to the bot to continue the conversation. On the other hand, Facebook Messenger provides deeper integration with systems such as Stripe and PayPal. In that version, the payment experience stays completely within the Facebook Messenger app. From a user perspective, the less friction the better. And as users begin placing more trust in messaging apps to store their payment information, we will see more and more payment integrations like this. Apple has released its Business Chat² product and you bet that Apple Pay payments are fully integrated.³

Common Enterprise Cases

Enterprise bots may be more specialized to a domain or subject matter. They are typically deployed using a web chat component or integrated into enterprise messaging systems, or even enterprise Call Center and Interactive Voice Response (IVR) systems such as Cisco's Unified Communications Center. They can also be deployed on e-mail endpoints. The bots may be integrated with single sign-on solutions, powerful existing enterprise back ends, and knowledge management databases. Depending on the enterprise’s practice, these will range from simple pilot bots to machine learning–driven large-scale deployments.

Self-Service Bots

One of the most common use cases in an enterprise scenario is incident self-servicing. Enterprises have large knowledge bases that internal help desk agents use to communicate possible solutions to the user and guide them through the process of troubleshooting issues. Many of these step-by-step troubleshooting directions can be communicated to the user by a bot. For example, one of the most common queries to internal help desks is password reset. Companies could cut through a lot of volume and, frankly, money, if they were to handle such requests automatically. You could imagine an appliance manufacturer releasing a chat bot to assist in diagnosis and fixing issues before involving a service engineer.

The idea behind these self-service bots is that they can provide a variety of self-service content for the users, especially for the most common queries, and can even automate some of the common work that the customer support team is doing. These bots are usually integrated with live chat systems so that the user may initially be chatting to a bot but can be quickly rerouted into a live chat or phone conversation with a human customer service agent in case the bot’s directions do not solve the problem.

Process Automation Bots

Robotic Process Automation (RPA) is a huge topic these days. Companies like IPsoft specialize in building bots and technology that can automate business and IT tasks.⁴ In this context, bots are not necessarily chat bots but rather computer agents that perform the automation. These tasks can include everything from account provisioning, website automation and business processes automation. There are companies that focus on creating automation platforms, such as Automation Anywhere and UiPath. With machine learning these days being used for everything from contract analysis to skin cancer diagnoses, chat bots can serve as an excellent front end into these processes. In an RPA scenario, the chat bot is more of an orchestrator rather than an automator. In addition, these bots may integrate into ticketing systems such as Remedy and ServiceNow to track its work.

In other instances, the chat bot is less visible to the user. For example, Slack is a great platform for bots that listen in to a team conversation and surface data as the right natural language arises. Bots that simply listen in to some natural language input and provide answers are a type of automation bot. Say, for example, a team of medical experts looks through text descriptions of procedures and is charged with translating them to insurance codes. That process can be automated by a machine learning algorithm that observes the team’s behavior and results and can then take over the data.

Again, the actual brains behind the logic may not be inside the bot itself. There may be a separate system that implements the machine learning model for the insurance codes. Or, the automation code may be Python, PowerShell, or any other script. The bot serves as the front end to receive the natural language and orchestrate the automation (Figure 4-7).

Knowledge Management Bots

Another type of enterprise bot is one that can solve natural language search problems across a variety of data sources. Many firms have huge knowledge repositories across disparate sets of systems. Being able to integrate with all those sources using natural language is important. There are interesting choices to make in these bots about which content to display to the user, in what format, and how to collect feedback on which content was the most useful given a query.

The bigger problems of natural language search that these projects try to solve are fascinating and beyond the scope of this book. This type of bot can be extremely interesting in a group conversation context, where the bot is querying articles, reports, white papers, and case studies as the group is having a conversation about topics of interest. The group’s feedback to the bot during the search can further provide supervised learning data to improve the search experience even further.

Representing Conversations

How do we start developing a conversational chat bot? A good place is trying to graphically represent the conversation flow. What kinds of tasks can the chat bot handle? What intents and entities does it need to look for to achieve these goals? How does it help fill in missing data?

We will be referring to the conversation as a graph, which is a collection of nodes connected by edges. Figure 4-8 illustrates an undirected graph. Every node is connected to at least one other node in the graph. Each node represents a state of the conversation, and the edges represent a transition between states.

We will use arrows in the edges to show the direction of flow. This is referred to as a directed graph . We start with the root node. The root node is the state of the beginning of the conversation. Using our Calendar bot as a sample, we know that our bot should support adding new entries, editing existing entries, removing entries, checking availability, and providing a summary of our calendar or an event. We can represent the bot as shown in Figure 4-9.

Note that the conversation moves between states based on the user’s utterance, which resolves to a LUIS intent. Each node along the conversation has built-in logic to resolve the entities and execute the correct logic for a state. After a state is done executing its logic, the conversation transfers back to the root node.

The transitions between states can be invoked either programmatically or by user input. For example, say our bot supports creating calendar appointments. Recall in Chapter 3 that we created a LUIS application that allows us to pass either several or no entities as part of an utterance to add a calendar entry. If our Add New Entry dialog did not receive information about a subject and invitee, as for example in the utterance “meet tomorrow at 2pm,” we could elicit that information in another state. On the other hand, if the user uses an utterance that contains these entities, such as “meet with kim for coffee tomorrow at 2pm,” we do not need to elicit this extra information. This conditional state transition is illustrated in Figure 4-10.

The process of creating a conversation graph is usually referred to as intent and entity mapping; we model intent and entity combinations as transitions between state nodes.

Bot Responses

There are a variety of forms that a bot response to a user’s query may take. Understanding the different options and how to best leverage them is key to any bot design. In the following sections we will dig into a number of concepts found amongst the various channels.

Building Blocks

We now understand how we can take user input and map it to bot states and function. We also understand how we can organize our bot code into various conversation states. The next step in our design is to figure out what the bot sends to the user in return. Bots can respond in a variety of ways. By default, we think of text or speech output. Most typically, we simply send back plain text. Some messaging channels support something more complex like Markdown or HTML. Markdown is a plain-text formatting syntax.⁵ The following Markdown input translates to the formatted content in Figure 4-11:

# H1

# H2

Hello, my _name_ is **Szymon Rozga**

I like:

1. Bots

1. Dogs

1. Music

Bot platforms can also support speech response. Many platforms also support the Speech Synthesis Markup Language (SSML) as a speech output format. SSML is a markup language that provides metadata about how speech should be constructed using elements such as pauses, breaks, changes of rate and pitch, and others. Here is a self-explanatory sample from the WC3 Recommendation⁶:

<?xml version="1.0"?>

<speak version="1.0" xmlns:="http://www.w3.org/2001/10/synthesis"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://www.w3.org/2001/10/synthesis

                   http://www.w3.org/TR/speech-synthesis/synthesis.xsd"

         xml:lang="en-US">

  That is a <emphasis> big </emphasis> car!

  That is a <emphasis level="strong"> huge </emphasis>

  bank account!

</speak>

Output to users does not always have to be text. We can use images and videos to communicate many ideas to our users. As part of any message sent back to the user, we may attach various content such as videos, audio files, and images. The specific supported formats will depend on the underlying operating system and channel. Some systems allow other file attachments as well, for instance XML files or a native format of some sort.

An alternative mechanism for presenting content to our users are cards. A card is typically a combination of an image, text, and optional buttons that serve as calls to action. Our YouTube Search bot from Chapter 1 (Figure 4-12) clearly displayed the video name, description, and a button to watch it in a set of cards.

This layout is called a carousel . It presents several cards side by side and gives the user the ability to swipe or scroll through the individual cards.

Buttons are typically sent as part of a card, but they can also be sent as stand-alone elements without an associate image. There are many types of buttons. The top three most popular buttons are used to open web pages, send a message back to the bot (IM back), or post a message back to the bot (post back). The difference between an IM back and a post back is that a post back message will not appear in the message history, whereas an IM back message would. Not all channels support both approaches, but the overall spirit of sending a message to the bot via a button click is widely supported.

Another type of button is a sign-in button. Sign-in buttons kick off an authentication or authorization flow via a login in a web view. Once the login is completed, the bot receives any necessary access tokens and can proceed with an authenticated session, as shown in Figure 4-13.

All the content described previously is kept within a user’s chat history. The carousels, the cards, the buttons, and, of course, all the text are available for the user to scroll through. There is one form of element that is displayed only in the context of the message that it is included in. That feature is suggested actions, also called quick replies . These buttons are presented on the bottom of the user interface until the user responds. These buttons are clear calls to actions and an indispensable tool for delightful conversational experiences. Figure 4-14 shows an example usage of suggested actions guiding users to video categories available in the TD Ameritrade bot .

Authentication and Authorization in Bots

Let’s be honest, no one is going to be sending a username and password to a bot chat window. This is a security risk. We do not want Facebook or Slack or any other channel to have our users’ login credentials in their message history. At the end of the day, a bot is simply a web service, so using the standard OAuth or OpenID Connect flows is a natural fit.

The right approach is to utilize a sign-in card, which is a card that includes a button that opens a login web page for the user to enter their credentials (Figure 4-15).

Typically, this login page will be an OAuth page (Figure 4-16).

OAuth 2.0⁷ is a standard for token-based authorization over the Internet. There are several different types of authorization flows that are enabled by OAuth 2.0. A three-legged OAuth flow allows a resource owner (the user) to grant access to an application (the consumer) to an API (the service provider). In the context of a bot, it looks as follows:

• The user clicks a button to open a login page to a service in a web view for the third party and enters their username/password combination. The URI for this login page typically includes a client ID and a redirect URI. The redirect URI is an endpoint on our bot web service.

• Once the user logs in successfully, the service redirects the user back to the bot redirect URI. The bot redirect URI endpoint receives the authorization code. This is the user’s grant to the application to use the service. The bot exchanges the authorization code for an access token (and an optional refresh token) from the token endpoint.

• The bot uses the access token when making requests to the service on behalf of the bot user.

• Typically, the access token is short lived, and the refresh token is longer lived. At any point, the bot can request a new access token from the token endpoint by posting the refresh token.

There is substantial documentation around the specifics of this and other OAuth flows. The RFC is a great starting point.⁸ The key point is that a bot is a web service, and the complete OAuth flows can happen in an integrated manner. The only tricky part from a UX perspective is to ensure that the browser window automatically closes when the login is completed. The various channels approach this in slightly different ways. Although one can implement the entire flow manually, something we show off in Chapter 8, the Bot Framework does provide additional tools to facilitate this process.⁹

Specialized Cards

On platforms that support them, cards are a key component of the user experience. We covered the idea of a generic cards. Some channels provide several specialized cards. For instance, a receipt card (Figure 4-17) can be sent to communicate a purchase receipt with information such as totals, tax, payment confirmation, and so forth.

In addition, Messenger gives developers the ability to utilize four air travel cards such as itinerary, boarding pass (Figure 4-18), check-in, and flight update.

Tapping the boarding pass card shows a full-screen version with a QR code that can be utilized at the airport (Figure 4-19). Depending on which platform we target, there may be other templates for us to use. If they exist and match your use cases, use them. They provide a good, native user experience.

Another form of specialized card is one in which you use custom graphics. A common approach is to generate the custom graphics on a web service as the bot processes user input. In Chapter 11, we will build a simple custom graphics renderer using Headless Chrome to show how easily we can begin building custom graphics using HTML and JavaScript.

Lastly, Microsoft has introduced a new card format called Adaptive Cards.¹⁰ Adaptive cards , which we will explode in Chapter 11, are a platform-agnostic manner to describe layouts of text, images, and input fields using a simple container-based layout engine. The Microsoft Bot channel connectors are then able to render the cards into platform-specific renderings. Adaptive Cards are a specialized version of the custom graphics approach integrated with logic to generate buttons and behavior in a card. It remains to be seen how many channels will end up supporting this format, but many of the Microsoft-owned channels already do.

Figure 4-20 shows an example of an HTML rendering of an Adaptive Card.

Figure 4-21 shows the rendering of an input form card on Microsoft’s Teams app.

Other Functions

Conversational Experience Guidelines

There are some key guidelines that we should follow when developing a bot experience. Some of them may not apply to every type of bot or may be more relevant to consumer versus enterprise bots, but one should keep this list in mind at a minimum when designing bots.

Focus

As discussed in Chapters 1 and 2, there are limits to the technology and how intelligent a bot can be. Our bots should not try to get too clever; humans will always be able to break the bot in one way or another. For example, it is quite OK to handle greetings from the user like “hi” or “hello. We do not want to go down the rabbit hole of being handling every different type of greeting. Don’t start creating specialized responses for “what’s up?” versus “hi.” If you are reading this book, you most likely don’t have the budget that Microsoft or Google has (Figure 4-22). We are here to help with tasks, not general AI. It is OK to be honest about our bot’s limitations.

Don’t Pretend the Bot Is a Human

We do not want our bots to end up in the uncanny valley.¹¹ That is, as with most, if not all, human-like objects, real humans will feel that something is not quite right, leading to strange and eerie feelings (Figure 4-23). We do not want our users to get those feelings. This goes hand in hand with representing your bot with human likeness. If you are representing your bot via an avatar, use an icon that clearly suggests a nonhuman entity. Siri and Cortana do this very well.

Always Present the Next Best Action

Our bot should never leave a user hanging without the user knowing what to do next. The bot should have a welcome message introducing itself, its capabilities, and some options to the user about what it is capable of. When the user is confused and asks for help or the bot is unable to recognize the user’s input, the bot should suggest some options as well. The key point is that if at any point of the conversation the user is met by a blank message box with no suggested next steps, it becomes a confusing conversational experience. Facebook Messenger, Skype, and other channels have a contextual quick-reply feature that presents button options of the bottom of the chat interface (Figure 4-24). Presenting such suggestions is a great way to communicate our bot’s capabilities and limitations.

Utilize Rich Content

Bots provide us with the opportunity to utilize more than just text. We can format text and include images, videos, and audio files. We can render cards (Figure 4-25) and even create some custom graphics in your cards. We need to utilize those features to their fullest extent

Provide a Clear Path to Humans

If there is one thing that should be clear by now, it is that bots cannot understand everything. Even with a limited scope of functions, there are going to be questions and issues that the bot will not be able to handle. Our bots should have an ability to somehow connect users to a human agent, if relevant to the use case. Whether it is displaying a phone number with a case number or having seamless integration into a live chat system, our bots should be clear in how our users can speak to humans for help with their issues (Figure 4-26). For example, I once encountered a bot that could answer frequently asked questions. I read a press release about the bot, so I decided to try it. I started the conversation and got a message about clicking a button. There were no buttons. I asked, “What can I do?” I got redirected to a human agent. At this point, I couldn’t do anything until a human dealt with my case. I also had no indication of how long it would take. Was their call center even open? Once the agent came around, I spoke to them and got sent back to the bot. I had total silence, no buttons. I said “Test.” The next message I got was that I was getting transferred again. At this point, I just quit. Don’t make your users quit in frustration.

Learn from Your Users

It is simple to use a conversational experience to collect data from users. It also easy to use user input in resolving conflicting intents from LUIS and then utilizing that data to train LUIS. Of course, the importance given to user input should be very different than the weight given to the utterances provided by a trainer. But if we have the data, we should use it to our advantage. Figure 4-27 shows an example of how we can implement such an approach. In the diagram we store user feedback into an active learning data store, and our active learning process determines how much of the same feedback it should observe before using the data point to train LUIS. Be careful with automated training based on user input. You do not want to go the way of Tay.¹³

There are more rules you may pick up on as you gain experience in this space across different messaging channels, but this list is a good starting point and something I suggest we follow on every chat bot project.

Conclusion

Conversation design is a rich field. We have numerous options for how we interact with users and how we communicate ideas in formats other than text. When developing bots, our approach should always be “do right by the user.” A user’s conversational experience can be very sensitive to tone, branding, verbosity, and overuse (you don’t need to use a card for everything). Although in the early phases there are some key abstractions, such as cards, the space has developed to best handle bot-to-user interactions. As bots become more commonplace, these mechanisms will improve and increase in number. Microsoft’s adaptive cards, for example, is a project that attempts to push the boundaries of what kind of functionality a bot can provide in conversation with users. My hope is that as bot become more and more commonplace, the messaging channels will support more and more types of behavior from bot cards.

We now have a good base understanding of the common operations bots perform and how they do so. The only remaining question is, how do we put all of this together in code? In the following chapter, we’ll do just that and put these ideas into practice.

Microsoft Bot Builder SDK Basics

The core library we will be using to write the bot is called the Bot Builder SDK ( https://github.com/Microsoft/BotBuilder ). To get started, you will need to create a new node package and install the botbuilder, dotenv-extended, and restify packages. You can do so by creating a new directory and typing these commands:

Figure 5-1 shows the typical high-level bot architecture on a local machine. The idea is that the node app relies, principally, on two components. First, the Bot Builder SDK is the bot engine we use to build our bots. Second, all messages from any channel, either external to the machine or the Bot Framework Emulator from the developer machine, are sent to the bot via an HTTP endpoint. We use restify to listen to HTTP messages and to send those to the SDK.

As an alternative to creating the package.json file manually, we can bootstrap this exercise using the echo-bot code provided with the book. The package.json for the echo-bot looks like this. Note that the eslint dependencies are purely for our development environment, so we can run a JavaScript linter¹ to check for stylistic and potential programmatic errors.

The bot itself is defined in the app.js file. Note that the start script in the package definition specifies app.js as the entry point of our bot.

Let’s walk thought this code. We use a library called dotenv to load environment variables.

The environment variables are loaded from a file called .env into the process.env JavaScript object. The .env.defaults file includes default environment variables and can be used to specify the values our Node.js requires. In this case, the file looks like this:

We require the botbuilder and restify libraries. Botbuilder is self-explanatory. Restify is used to run a web server endpoint for us.

Now we set up our web server to listen for messages on port 3978.

Next, we create what is called a chat connector . In the context of the Bot Framework, the channel connectors are endpoints created and maintained by Microsoft that help translate messages from the native platform format to the Bot Builder SDK format . The builder.ChatConnector object knows how to receive HTTP messages from these connectors, pass them to the bot conversation engine, and send any outgoing messages back to the connectors, as in Figure 5-2.

The environment variables MICROSOFT_APP_ID and MICROSOFT_APP_PASSWORD are our bot’s credentials. We will set them up in the Bot Framework at a later point when we create the Azure Bot Service registration with Azure. For now, we can leave those values blank because we don’t care to secure our bot quite yet.

Next we tell restify that any requests into the /api/messages endpoint, or, more specifically, http://localhost:3978/api/messages, should be handled by the function returned by connector.listen(). That is, we are allowing the Bot Framework to handle all incoming messages into that endpoint .

Lastly, we create the universal bot. It is called a universal bot because it is not tied to any specific platform. It uses the connector for receiving and sending data. Any message that comes into the bot will be sent to the array of functions. For now, we have only one function. The function takes in a session object. This object contains data such as the message but also data about the user and conversation. The bot responds to the user by calling the session.send function.

Notice that the Bot Builder SDK takes care of providing the right HTTP response to the incoming HTTP request. In practice, the internals will return an HTTP Accepted (202) if Bot Builder processes the code without a problem and will return an HTTP Internal Server Error (500) otherwise.

The content of our response is asynchronous, meaning that the response to the original request our bot receives does not contain any content. An incoming request, as we will see in the following chapter, includes a channel ID, the name of the connector such as slack or facebook, and a response URL where our bot sends messages. The URL typically looks like https://facebook.botframework.com . Session.send will send an HTTP POST request to the response URL.

We can run this bot by simply executing the following:

We will see some Node.js output in our console. There should be a server running on port 3978, on the path /api/messages. Depending on our local Node.js setup and the preexisting software on our machine, we may need to update to the latest version of the node-gyp package, a tool for compiling native addon tools.

How do we actually converse with the bot? We could try sending messages by using a command-line HTTP tool like curl, but we would have to host a response URL to see any responses. Additionally, we would need to add logic to obtain an access token to pass any security checks. Seems like too much work to simply test the bot.

Of course, we do not have to do any of this. Microsoft provides an emulator for us to test our bots. It is available at https://emulator.botframework.com/ for download. The emulator supports Linux, Windows, and OS X (Figure 5-3).

Let’s go ahead and connect to our bot. We enter http://localhost:3978/api/messages into the address bar and leave the Microsoft App ID and Microsoft App Password fields empty for now (Figure 5-4) since we haven’t set up a .env file. We will receive security warnings in the console; these are fine to ignore for now. At this point, we are really to click the Connect button.

We will see two messages appear in the emulator log. Both are of type conversationUpdate (Figure 5-5).

What does this mean? Each message between the bot and consuming connector (Emulator in this case) is called an activity, and each activity has a type. There are types like message or typing. If the activity is of type message, then it is literally a message between the bot and the user. A typing activity tells the connector to display a typing indicator. Previously, we saw the conversationUpdate type. This type indicates there is a change in the conversation; most commonly, users have joined or left the conversation. In a 1:1 conversation between a user and a bot, the user and bot will be the two members of a conversation. In a group chat scenario, the bot plus all the users would be part of the conversation. The message metadata will include information about which users joined or left the conversation. In fact, if we click the POST link for the two conversationUpdate activities, we find the JSON in the Details section. Here is the content for both messages:

Now, let’s send a message to the bot with the text “echo!” and look at the Emulator logs (Figure 5-6). Note that if we do not set up an explicit bot storage implementation, we might get a warning like this: “Warning: The Bot Framework State API is not recommended for production environments, and may be deprecated in a future release.” We will dive into this in the next chapter. Suffice it to say, it is strongly suggested that we use do not use the default bot storage. We can use the following code to replace it for now:

const inMemoryStorage = new builder.MemoryBotStorage();

bot.set('storage', inMemoryStorage);

Let’s examine the messages exchanged in more detail. The incoming message from the emulator looks as follows:

There should be no surprises here. The response looks similar though a lot less verbose. This is typical. The incoming message will be populated by the channel connector with as much supporting data as possible. The response does not need to have all of this. One item of note is the fact that ID is not populated; the channel connector will typically take care of this for us.

We also note the existence of the inputHint field, which is mostly relevant to a voice assistant system and is an indication to the messaging platform on the suggested state for the microphone. For example, acceptingInput would indicate the user may respond to a bot message, and expectingInput would indicate that a user response is expected right now.

Lastly, the Debug Event provides data on how the bot executed the request.

Note, these are the same values as are printed in the bot console output. Again, if we did not override the default bot state, we would see more data here related to the deprecated code. The console output is shown here:

This output tracks how the user’s request is executed and how it traverses the conversation dialogs. We will address this further in this chapter.

If we were to send more messages using the emulator, we would see the same type of output since this bot is very simple. As we gain more experience with features such as cards, we will benefit from using the Emulator and examining the JSON messages further. The protocol is a huge part of the Bot Framework’s power: we should be as familiar with it as we can.

Bot Framework End-to-End Setup

We now have a bot. How do we connect it to all these different channels? The Bot Framework makes this simple. Our goal here will be to register our bot and its endpoint with the Bot Framework through the Azure Portal and subscribe the bot to the Facebook Messenger channel.

There are a few things we will have to do. First, we have to create an Azure Bot service registration on the Azure Portal. We may need to create our first Azure Subscription. Part of this setup is using ngrok to allow the bot to be accessible from the Internet, so we should make sure that we have ngrok installed from here: https://ngrok.com/ . Lastly, we will deploy the bot to Facebook Messenger. This means we need to create a Facebook page, a Facebook app, and Messenger and Webhook integrations, and connect all that back to the Bot Framework. There are quite a few steps, but once we become familiar with Azure and Facebook terminology, it is not that cumbersome. We will first quickly walk through the directions and then go back and explain what was done at each step.

Step 1: Connecting to Azure

Our first step is to log into the Azure Portal. If you have an Azure account, excellent. Skip ahead to step 2 if you have an Azure subscription already. If you do not, you are able to create a free developer account with a $200 30-day credit by going to https://azure.microsoft.com/en-us/free/ .

Click “Start free.” You will need to log in using a Microsoft or work account. If you have neither, you can easily create a Microsoft account at https://account.microsoft.com/account . Once you’re authenticated, you will see a page like in Figure 5-7. This page will collect your personal information and verify your identity via a text message and a valid credit card. Don’t be alarmed. The credit card is necessary to verify your identity. Chances are, you will not come even close to using the $200 credit, and if you do, you will not be charged; you will just not be able to use the services further. Much of what we use Azure for in this book can be accommodated via a free tier of the various Azure services.

Once the process is done, you will be able to go into the Azure Portal at https://portal.azure.com . It looks something like Figure 5-8. On the top right, you will see the email address you signed up with and your directory name. For example, if my email were szymon.rozga@aol.com (it’s not), then my directory name would be SZYMONROZGAAOL. If you were added into other directories, that menu would be a drop-down for you to select to which directory you are navigating.

An Azure account contains subscriptions. A subscription is a billing entity. If we navigate to https://portal.azure.com/#blade/Microsoft_Azure_Billing/SubscriptionsBlade , or the Subscriptions service in the portal, and we had just created the $200 trial account, we should see one subscription with the name Free Trial. Each Azure subscription can also contain one or more resource groups. A resource group is a logical container for resources, which are individual Azure services. All costs associated with resources in each resource group are charged against the payment method associated with the containing subscription. With the $200 trial account, services are automatically shut down when the combined costs reach the spending limit. If desired, the free account can be converted to a paid account that will accrue additional charges to your credit card (or alternative payment options).

Step 2: Creating the Bot Registration

In the Azure Portal, click the “Create a resource” button in the top-left pane. In the Search the Marketplace text field, enter azure bot. You will get many results, but we are interested in the top three (Figure 5-9).

These are the three options:

• Web App Bot: A bot registration pointing to a web app deployed on Azure

• Functions Bot: A bot registration pointing to a bot running as an Azure function, one of Azure’s serverless computing options

• Bot Channels Registration: A bot registration with no cloud-based back end

For our purposes, we will create a Bot Channels Registration bot, as we will keep on running the bot locally on our laptop. Click Bot Channels Registration and then click Create. As per Figure 5-10, enter a bot name, the name for the resource group that will contain this registration, and the resource location, namely, the Azure region that will host the registration. For Pricing Tier, select F0; this is the free option and sufficient for our needs. Leave the messaging endpoint empty for now and leave Application Insights selected to On. Application Insights is one of Microsoft’s cloud telemetry and logging services. The Bot Framework uses this to store data and analytics on your bot registration usage. By default, this will create the basic and free tier of Application Insights. Select as close a location to the Bot Channels Registration location as possible. Click Create when ready.

There is a progress indicator across the top of the portal, and we will receive a notification when the registration is ready. We can also navigate into the resource group by using the Resource Groups button on the left pane (Figure 5-11).

Navigate into the bot channel registration and then navigate to the Settings blade (Figure 5-12). Note that Azure automatically populated the Application Insights identifiers and keys. These will be used to track analytics data for our bot. We will see one of the resulting analytics dashboards in Chapter 13.

We will also be shown the Microsoft App ID. Take note of this value. Click the Manage link directly above it to navigate to the Microsoft Application Portal. This may ask for our login information once again because it is a separate site from Azure. Once we locate the newly created bot in the list of applications, click Generate New Password (in the Application Secrets section) and save the value; you will see it once only! Recall that we were seeing warnings that our bot was not secure in the bot console output? We will now fix that.

Step 4: Setting Up Remote Access

We could deploy the bot to Azure, connect the Facebook connector to that endpoint, and call it a day. But how do we develop or debug Facebook-specific features? The Bot Framework way is to run a local instance of the bot and connect a test Facebook page to the local bot for development.

To achieve this, run ngrok from the command line.

ngrok http 3978

We will be presented with the data in Figure 5-13. By default, ngrok assigns a random subdomain (paid ngrok versions allow you to specify a domain name). In this case, my URL is https://cc6c5d5f.ngrok.io . Note that the free version of ngrok provides a random subdomain each time we run it. We can get around this by either upgrading to a paid version or simply leaving the ngrok session up for as long as possible.

Let’s see if this works. In the emulator, enter the ngrok URL followed by /api/messages. For instance, for the previous URL, the correct messaging endpoint is https://cc6c5d5f.ngrok.io/api/messages . Add the app ID and app password information into the emulator. Once you click Connect, the emulator should successfully connect to and chat with the bot.

Now, assign the same messaging endpoint URL in the Bot Channels Registration Settings blade, Figure 5-12. Next, navigate to the Test with Web Chat blade and try to send a message to the bot. It should work. You’ve connected your first channel to your bot (Figure 5-14)!

Step 5: Connecting to Facebook Messenger

Pretty cool, right? The Bot Framework is almost completely integrated with our bot. We will now proceed to integrate our bot with Facebook Messenger. The Channels blade on the Bot Channels Registration gives us an ability to connect to the Microsoft-supported channels (Figure 5-15).

Click the Facebook Messenger button to go into the Messenger configuration screen (Figure 5-16). We are going to need to get four pieces of data from Facebook: page ID, app ID, app secret, and page access token. Lastly, we should take note of the callback URL and verify token. We will need these to set up connectivity between Facebook and the Bot Framework.

Let’s now set up the necessary Facebook assets. We must have a Facebook account to complete the following tasks. Navigate to Facebook.com and use the top-right drop-down menu to create a new page (Figure 5-17). Facebook will ask for the type of page. For the purposes of this example, we can select the Brand/Product type and App Page subcategory.

I created a page called Szymon Test Page. We can find the page ID by clicking to the About link on the left navigation pane (Figure 5-18). On the very bottom we will find the page ID. We need to copy that value into the Bot Framework Facebook Messenger channel configuration form (Figure 5-16).

Next, in a new browser tab or window, navigate to https://developers.facebook.com . If you have not yet done so, register for a developer account. Create a new app (Figure 5-19). Give it any name you like.

Once you do, navigate to the Settings ➤ Basic page via the left sidebar menu and copy the Facebook app ID and app secret into the Bot Framework form (Figure 5-20).

Next, navigate to the Dashboard (from the link on the left sidebar) and set up the Messenger product. Scroll down the page until you get to the Token Generation section. Generate a page access token by selecting the page in the Token Generation section (Figure 5-21). Copy the token into the Bot Framework form within the Azure Portal.

Next, scroll to the Webhooks section (just below the Token Generation section of the Facebook App Dashboard) and click Setup Webhooks. You will see a pop-up that asks you for a callback URL and the verify token. Copy and paste both of those from the Configure Facebook Messenger form in the Azure Portal.

In the Subscription Fields section, select the following fields:

• messages

• message_deliveries

• message_reads

• messaging_postbacks

• messaging_optins

• message_echoes

Click Verify and Save. Lastly, select the page you would like your bot to subscribe to from the drop-down and click Subscribe. Your setup page should look as shown in Figure 5-22.

Make sure to save the Bot Framework configuration. That’s it! You can find the page in your Messenger contacts. You can send it a message, and you should get it echoed back (Figure 5-23).

Step 6: Deploying to Azure

It would not be a complete tutorial if we did not deploy the code into the cloud. We will create a web app and deploy our Node.js app using Kudu ZipDeploy. Lastly, we will point the bot channel registration to the web app.

Go into your Azure resource group that we created in step 2 and create a new resource. Search for web app. Select Web App and not Web App Bot. The Web App Bot is a combination of a bot channel registration and an app service. We have no need for this combination since we have already created a bot channel registration.

When creating the web app, we will need to give it a name. Also ensure the correct resource group is selected (Figure 5-24). Azure will add it to our existing resource group and create a new app service plan for us. An app service plan is a container for web apps and similar compute resources; it defines the hardware on which our apps run as well as the costs. In Figure 5-24, we create a new app service plan and choose the Free pricing tier. Free is good.

Before we deploy our echo bot, we need to add two things. First, we add a response to the base URL endpoint to validate our bot was deployed. Add this code to the end of the app.js file:

server.get('/', (req, res, next) => {

    res.send(200, { "success": true });

    next();

});

Second, for a Windows-based Azure setup, we also need to include a custom web.config file to tell Internet Information Services (IIS)² how to run a Node app.³

<?xml version="1.0" encoding="utf-8"?

<!--

     This configuration file is required if iisnode is used to run node processes behind

     IIS or IIS Express.  For more information, visit:

     https://github.com/tjanczuk/iisnode/blob/master/src/samples/configuration/web.config

-->

<configuration>

  <system.webServer>

    <!-- Visit http://blogs.msdn.com/b/windowsazure/archive/2013/11/14/introduction-to-websockets-on-windows-azure-web-sites.aspx for more information on WebSocket support -->

    <webSocket enabled="false" />

    <handlers>

      <!-- Indicates that the server.js file is a node.js site to be handled by the iisnode module -->

      <add name="iisnode" path="app.js" verb="*" modules="iisnode"/>

    </handlers>

    <rewrite>

      <rules>

        <!-- Do not interfere with requests for node-inspector debugging -->

        <rule name="NodeInspector" patternSyntax="ECMAScript" stopProcessing="true">

          <match url="^app.js\/debug[\/]?" />

        </rule>

        <!-- First we consider whether the incoming URL matches a physical file in the /public folder -->

        <rule name="StaticContent">

          <action type="Rewrite" url="public{REQUEST_URI}"/>

        </rule>

        <!-- All other URLs are mapped to the node.js site entry point -->

        <rule name="DynamicContent">

          <conditions>

            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="True"/>

          </conditions>

          <action type="Rewrite" url="app.js"/>

        </rule>

      </rules>

    </rewrite>

    <!-- 'bin' directory has no special meaning in node.js and apps can be placed in it -->

    <security>

      <requestFiltering>

        <hiddenSegments>

          <remove segment="bin"/>

        </hiddenSegments>

      </requestFiltering>

    </security>

    <!-- Make sure error responses are left untouched -->

    <httpErrors existingResponse="PassThrough" />

    <!--

      You can control how Node is hosted within IIS using the following options:

        * watchedFiles: semi-colon separated list of files that will be watched for changes to restart the server

        * node_env: will be propagated to node as NODE_ENV environment variable

        * debuggingEnabled - controls whether the built-in debugger is enabled

      See https://github.com/tjanczuk/iisnode/blob/master/src/samples/configuration/web.config for a full list of options

    -->

    <!--<iisnode watchedFiles="web.config;*.js"/>-->

  </system.webServer>

</configuration>

Next, we visit our bot web app via the browser. In my case, I navigate to https://srozga-test-bot-23.azurewebsites.net . There will be a default “Your App Service app is up and running” page. Before we deploy, we must zip the echo bot for transfer to Azure. We zip all the application files, including the node-modules directory. We can use the following commands:

# Bash

zip -r echo-bot.zip .

# PowerShell

Compress-Archive -Path * -DestinationPath echo-bot.zip

Now that we have a zip file, we have two options as to how we deploy. In option 1, we use a command line to deploy the bot by using the Kudu⁴ endpoint at https://{WEB_APP_NAME}.scm.azurewebsites.net. To enable this, we must first visit the Deployment Credentials blade in the app service (Figure 5-25) to set up a deployment username and password combination.

Once done, we are ready to roll. Running the following curl command will kick off the deployment process:

curl -v POST -u srozga321 --data-binary @echo-bot.zip https://srozga-test-bot-23.scm.azurewebsites.net/api/zipdeploy

Once you run this, curl will ask for the password that was provided in Figure 5-25. It will upload the zip and set up the app on the app service. Once done, make a request to your app’s base URL, and you should see a 200 response with success set to true.

$ curl -X GET https://srozga-test-bot-23.azurewebsites.net

{"success":true}

The alternative way to deploy is to use the Kudu interface on the SCM website: https://srozga-test-bot-23.scm.azurewebsites.net/ZipDeploy . You can simply drag and drop your zip file on the file listing in Figure 5-26.

There’s one more step. Go into the Settings blade in the Bot Channels Registration entry and set the messages endpoint setting to your new app service (Figure 5-27). Make sure to click the Save button.

Save and test on Web Chat and Messenger to your heart’s content. Congratulations! We accomplished a lot! We now have a bot running on Azure using Node.js and the Microsoft Bot Framework talking to Web Chat and Facebook Messenger. Up next, we will dive into a description of what we just accomplished.

What Did We Just Do?

We went through quite a lot in the previous section. There are many moving parts in terms of registering and creating a bot, establishing connectivity to Facebook and deploying to Azure. Many of these action only need to be performed once, but as a bot developer you should have a solid understanding of the different systems, how they connect to each other and how they can be set up.

Bot Channels Registration Entry

When we create the bot channels registration, we are creating a global registration that can be used by all of the channel connectors to identify, authenticate, and communicate with our bot. Each connector, whether it communicates to Messenger, Slack, Web Chat, or Skype, knows about our bot, its Microsoft app ID/password, the messages endpoint, and other settings (Figure 5-28). The bot channels registration is the starting point for Bot Framework bots.

We skipped the other two types of bot resources in Azure: Web App Bot and Functions Bot. A Web App Bot is exactly what we just set up; we provision a server to run bot app. Azure Functions is one of Azure’s approaches to serverless computing. It allows us to host different code, or functions, in a cloud environment to be run on demand. We pay only for the resources we use. Azure scales the infrastructure dynamically based on load. Functions is a perfectly valid approach to bot development. For more complex scenarios, we need to be careful about architecting the function code for scale-out and multiserver deployment. For the purposes of this book, we do not utilize Functions Bots. However, we suggest you experiment with the topic as serverless computing is becoming more and more prominent.

Key Bot Builder SDK Concepts

It feels good to have worked through the details of getting a bot running via the emulator and Facebook Messenger, but the bot doesn’t do anything useful! In this section, we will delve into the Bot Builder SDK for Node.js library. This is the focus for the rest of this chapter and the following chapter. For now, we will go over four foundational concepts of the Bot Builder SDK. Afterward, we show the skeleton code for a calendar bot conversation, based on the NLU work on LUIS from Chapter 3. This bot will know how to talk to users about many calendar tasks but will not integrate with any APIs quite yet. This is a common approach to demonstrate the conversation flow and how it may work without going through the entire back-end integration effort. Let’s dive in.

Sessions and Messages

Session is an object that represents the current conversation and the operations that can be invoked on it. At the most basic level, we can use the Session object to send messages.

const bot = new builder.UniversalBot(connector, [

    session => {

        // for every message, send back the text prepended by echo:

        session.send('echo: ' + session.message.text);

    }

]);

Messages can include images, videos, files, and custom attachment types. Figure 5-29 shows the resulting message.

session => {

    session.send({

        text: 'hello',

        attachments: [{

            contentType: 'image/png',

            contentUrl: 'https://upload.wikimedia.org/wikipedia/commons/b/ba/New_York-Style_Pizza.png',

            name: 'image'

        }]

    });

}

We also can send a hero card. Hero cards are stand-alone containers that include an image, title, subtitle, and text plus an optional list of buttons. Figure 5-30 shows the resulting exchange.

let msg = new builder.Message(session);

msg.text = 'Pizzas!';

msg.attachmentLayout(builder.AttachmentLayout.carousel);

msg.attachments([

    new builder.HeroCard(session)

        .title('New York Style Pizza')

        .subtitle('the best')

        .text("Really, the best pizza in the world.")

        .images([builder.CardImage.create(session, 'https://upload.wikimedia.org/wikipedia/commons/b/ba/New_York-Style_Pizza.png')])

        .buttons([

            builder.CardAction.imBack(session, "I love New York Style Pizza!", "LOVE THIS")

        ]),

    new builder.HeroCard(session)

        .title('Chicago Style Pizza')

        .subtitle('not bad')

        .text("some people don't believe this is pizza.")

        .images([builder.CardImage.create(session, 'https://upload.wikimedia.org/wikipedia/commons/3/33/Ginoseastdeepdish.jpg')])

        .buttons([

            builder.CardAction.imBack(session, "I love Chicago Style Pizza!", "LOVE THIS")

        ]),

]);

session.send(msg);

This example introduces some new concepts. The hero card is just one type of card that the Bot Builder SDK supports. The following are other supported cards:

• Adaptive card : A flexible card with a combination of items including containers, buttons, input fields, speech, text, and images; not supported by all channels. We dive into adaptive cards in Chapter 11.

• Animation card : A card that supports animated GIFs or short videos.

• Audio card: A card to play audio.

• Thumbnail card: Similar to a hero card but with a smaller image size.

• Receipt card : Renders a receipt including common line items such as description, tax, totals, etc.

• Sign in card: A card to initiate a sign-in flow.

• Video card: A card to play videos.

Another interesting point is the attachment layout. By default, attachments are sent in a vertical list. We chose to use the carousel, a scrollable horizontal list, to provide a better experience for the user.

The buttons in this code use the IM Back action. This sends the button’s value field (“I love New York Style Pizza!” or “I love Chicago Style Pizza!”) as a text message to the bot when the LOVE THIS buttons are clicked. Other action types are described below. Each messaging platform has different levels of support for these types.

• postBack: Same as IM back, but the user doesn’t see the message.

• openUrl: Opens a URL in a browser. This can be the default browser on the desktop or an in-app web view.

• call: Calls a phone number.

• downloadFile: Downloads a file to the user’s device.

• playAudio: Plays an audio file.

• playVideo: Plays a video file.

• showImage: Shows an image in an image viewer.

We can also use the Session object to send speech consent in channels that support both written and spoken responses. We can either build a message object like we did in the carousel hero card sample or use a convenience method on the session. The input hint in the following code snippet tells the user interface whether the bot is expecting a response, accepting input, or not accepting input at all. For developers who have a background in voice assistant skill development, like for Amazon’s Alexa, this should be a familiar concept.

const bot = new builder.UniversalBot(connector, [

    session => {

        session.say('this is just text that the user will see', 'hello', { inputHint: builder.InputHint.acceptingInput});

    }

]);

Session is also the object that helps us access relevant user conversation data. For example, we can store the last message the user sent to the bot inside the session’s privateConversationData and utilize it later in the conversation as shown in the following sample (Figure 5-31):

session => {

    var lastMsg = session.privateConversationData.last;

    session.privateConversationData.last = session.message.text;

    if(lastMsg) {

        session.send(lastMsg);

    } else {

        session.send('i am memorizing what you are saying');

    }

}

The Bot Builder SDK makes it easy to store three types of data in the session object.

• privateConversationData: Private user data scoped to a conversation

• conversationData: Data for a conversation, shared between all users who are part of the conversation

• userData: Data for a user across all conversations on one channel

By default, these objects are all stored in memory, but we can easily provide an alternate storage service implementation. We will see an example in Chapter 6.

Waterfalls and Prompts

A waterfall is a sequence of functions that process incoming messages on a bot. The Universal Bot constructor takes an array of functions as a parameter. This is the waterfall. The Bot Builder SDK calls each function in succession, passing the result of the previous step into the current step. The most common use of this approach is to query the user for more information using a prompt. In the following code, we use a text prompt, but the Bot Builder SDK supports inputs such as numbers, dates, or multiple choice (Figure 5-32).

const bot = new builder.UniversalBot(connector, [

    session => {

        session.send('echo 1: ' + session.message.text);

        builder.Prompts.text(session, 'enter for another echo!');

    },

    (session, results) => {

        session.send('echo 2: ' + results.response);

    }

]);

We can also advance the waterfall manually, using the next function, in which case the bot would not wait for additional input (Figure 5-33). This is useful in cases where the first step may conditionally ask for additional input. We will use this in our calendar bot code.

const bot = new builder.UniversalBot(connector, [

    (session, args, next) => {

        session.send('echo 1: ' + session.message.text);

        next({response: 'again!'});

    },

    (session, results, next) => {

        session.send('echo 2: ' + results.response);

    }

]);

The following is an even more complex data-gathering waterfall:

const bot = new builder.UniversalBot(connector, [

    session => {

        builder.Prompts.choice(session, "What do you want to do?", "add appointment|delete appointment", builder.ListStyle.button);

    },

    (session, results) => {

        session.privateConversationData.action = { type: results.response.index };

        builder.Prompts.time(session, "when?");

    },

    (session, results, next) => {

        session.privateConversationData.action.datetime = results.response.resolution.start;

        if (session.privateConversationData.action.type == 0) {

            builder.Prompts.text(session, "where?");

        } else {

            next({ response: null });

        }

    },

    (session, results, next) => {

        session.privateConversationData.action.location = results.response;

        let summary = null;

        const dt = moment(session.privateConversationData.action.datetime).format('M/D/YYYY h:mm:ss a');

        if (session.privateConversationData.action.type ==  0) {

            summary = 'Add Appointment ' + dt + ' at location ' + session.privateConversationData.action.location;

        } else {

            summary  = 'Delete appointment  ' + dt;

        }

        const action = session.privateConversationData.action;

        // do something with action

        session.endConversation(summary);

    }

]);

In this sample, we use a couple more types of prompts: Choice and Time. The Choice prompt asks the user to select an option. The prompt can render the choice using inline text (relevant in SMS scenarios for example) or buttons. The Time prompt uses the chronos Node.js library to parse a string representation of a datetime into a datetime object. An input like “tomorrow at 5pm” can resolve to a value that a computer can use.

Note that we use logic to skip certain waterfall steps. Specifically, if we are in the delete appointment branch, we do not need the event location. As such, we do not even ask for it. We take advantage of the privateConversationData object to the store action object, which represents the operation we will want to invoke against an API. Lastly, we use the session.endConversation method to finalize the conversation. This method will clear the user’s state so that the next time the user interacts with the bot, it is as if the bot is seeing a new user.

Figure 5-34 shows the resulting conversation.

Dialogs

Let’s bring this full circle with conversational design. In Chapter 4, we discussed how we can model a conversation using a graph of nodes we called dialogs. So far in this chapter, we have learned about waterfalls and how we can model a conversation in code.

We have also learned how we can utilize prompts to gather data from the user. Recall that prompts are simple mechanisms to collect data from users.

builder.Prompts.text(session, "where?");

Prompts are interesting. We call a function (builder. Prompts.text), yielding the conversation to the prompt. Once a valid response is sent by the user, the next step in our waterfall can access the prompt’s result. Figure 5-35 shows the overall process. From our waterfall’s perspective, we don’t really know what the Prompts.choice call is doing, nor do we care. It is listening for user input, doing some validation, reprompting on bad input, and returning only a valid result, unless the user cancels. All that logic is hidden from us.

This interaction is the same model as a programming function call. The way a function call is typically implemented is using a stack. Examine Figure 5-36 and the following code:

function f(a,b) { return a + b; }

When the function f is called, the function’s arguments are pushed on the top of the stack. The function’s code then processes the stack. In this example, the function adds the parameters. At the end, the only value left at the top of the stack is the function’s return value. The calling function can then do whatever it wants with the return value.

This is the way prompts work in a conversation. The generic concept in the Bot Builder SDK is a dialog. A prompt is a type of dialog. A dialog is nothing more than an encapsulation of conversation logic and is analogous to a function call. A dialog is initialized with some parameters. It receives input from the user, executing its own code or calling into other dialogs along the way, and can send responses back to the user. Once the dialog’s purpose is accomplished, it returns a value to the calling dialog. In short, a calling dialog pushes a child dialog to the top of the stack. When the child dialog is done, it pops itself from the stack.

Let’s go back to our Choice prompt example. In the dialog stack model, the Root dialog places the Prompt.Choice dialog on the top of the stack. After the dialog finishes executing, the resulting user input object is passed back down to the Root dialog. The Root dialog then does whatever it needs to do with the resulting object. The behavior over time is captured in Figure 5-37.

We could take this concept even further. We could imagine a flow in our calendar bot in which adding a new calendar entry invokes a new dialog. Let’s call it AddCalendarEntry. It then invokes a Prompt.Time dialog to gather the date and time of the event, and it invokes a Prompt.Text dialog to gather the event’s subject. The AddCalendarEntry packages the collected data and creates a new calendar entry by calling some calendar API. Control is then returned to the Root dialog. We illustrate this in Figure 5-32. We could even have AddCalendarEntry call another dialog that encapsulates the logic to call the API if there was enough complexity in that process and we wanted to reuse the logic from other dialogs (Figure 5-38).

Waterfalls and dialogs are the workhorses that translate a conversation design into actual working code. There are, of course, more details around them, and we’ll get into those during the next chapters, but this is the magic behind the Bot Builder SDK. Its key value is an engine that can drive a conversation using the dialog abstractions. At each point during the conversation, the dialog stack and the supporting user and conversation data are stored. This means that depending on the conversation’s storage implementation, the user may stop talking to the bot for days, come back, and the bot can pick up from where the user left off.

How do we apply some of these concepts? Revisiting the add and remove appointment waterfall sample, we can create a bot that, based on a Choice prompt, starts one of two dialogs: one to add a calendar entry or another to remove it. The dialogs have all the necessary logic to figure out which appointment to add or remove, resolve conflicts, prompt for user confirmation, and so forth.

const bot = new builder.UniversalBot(connector, [

    session => {

        builder.Prompts.choice(session, "What do you want to do?", "add appointment|delete appointment", builder.ListStyle.button);

    },

    (session, results) => {

        if (results.response.index == 0) {

            session.beginDialog('AddCalendarEntry');

        } else if (results.response.index == 1) {

            session.beginDialog('RemoveCalendarEntry');

        }

    },

    (session, results) => {

        session.send('excellent! we are done!');

    }

]);

bot.dialog('AddCalendarEntry', [

    (session, args) => {

        builder.Prompts.time(session, 'When should the appointment be added?');

    },

    (session, results) => {

        session.dialogData.time = results.response.resolution.start;

        builder.Prompts.text(session, 'What is the meeting subject?');

    },

    (session, results) => {

        session.dialogData.subject = results.response;

        builder.Prompts.text(session, 'Where should the meeting take place?');

    },

    (session, results) => {

        session.dialogData.location = results.response;

        // TODO: take the data and call an API to add the calendar entry

        session.endDialog('Your appointment has been added!');

    }]);

bot.dialog('RemoveCalendarEntry', [

    (session, args) => {

        builder.Prompts.time(session, 'Which time do you want to clear?');

    },

    (session, results) => {

        var time = results.response.resolution.start;

        // TODO: find the relevant appointment, resolve conflicts, confirm prompt, and delete

        session.endDialog('Your appointment has been removed!');

    }]);

We start a new dialog by calling the session.beginDialog method and pass in the dialog name. We may also pass an optional argument object, which would be accessible via the args parameter in the called dialog. We use the session.dialogData object to store dialog state. We’ve run into userData, privateConversationData, and conversationData before. Those are scoped to the entire conversation. DialogData, however, is scoped only to the lifetime of the current dialog instance. To end a dialog, we call session.endDialog. This returns control to the next step in the root waterfall. There is a method called session.endDialogWithResult that allows us to pass data back to the calling dialog.

The conversation in Messenger ends up looking like Figure 5-39.

This code has a few shortcomings. First, if we want to cancel either adding or deleting an appointment, there’s no way to do that. Second, if we are the middle of adding an appointment and decide we want to delete an appointment, we cannot easily switch to the remove appointment dialog. We must finish the current dialog and then switch over. Third, but not essential, it would be nice to connect the bot to our LUIS model so users can interact with the bot using natural language. We’ll address the first two points next and then follow with connecting to our LUIS models to really build some intelligence into the bot.

Invoking Dialogs

Let’s continue with the following exercise. Say we want to allow the user to ask for help at any point in the conversation; this is a typical scenario. Sometimes the help will be contextual to the dialog. At other times, the help will be a global action, a bot behavior that can be accessed from anywhere within the conversation. The Bot Builder SDK allows us to insert both types of behaviors into our dialogs.

We introduce a simple help dialog.

bot.dialog('help', (session, args, next) => {

    session.endDialog("Hi, I am a calendar concierge bot. I can help you make and cancel appointments!");

})

.triggerAction({

    matches: /^help$/i

});

This code defines a new dialog with a global action handler that matches the “help” input. TriggerAction defines a global action. We are saying that the help dialog will be triggered globally whenever the user’s input matches the regular expression ^help$. The ^ character denotes the start of a line, and the $ character denotes the end of a line. A problem arises, though. As we can see in Figure 5-40, it looks as if when we ask for help, our bot forgets we were in the add appointment dialog. In fact, the default behavior for global action matching is to replace the dialog on top of the stack. In other words, the add appointment dialog was removed and replaced with the help dialog.

We can override this behavior by implementing the onSelectAction callback.

bot.dialog('help', (session, args, next) => {

    session.endDialog("Hi, I am a calendar concierge bot. I can help you make and cancel appointments!");

})

.triggerAction({

    matches: /^help$/i,

    onSelectAction: (session, args, next) => {

        session.beginDialog(args.action, args);

    }

});

This brings up an interesting question: how can we affect the dialog stack? When we are working on a dialog flow and want to transition control over to another dialog, we can use either beginDialog or replaceDialog. replaceDialog replaces the dialog on top of the stack, and beginDialog pushes a dialog to the top of the stack. The session also has a method called reset, which resets the entire dialog stack. The default behavior is to reset the stack and push the new dialog on top.

What if we wanted to include contextual help? Let’s create a new dialog to handle help for the add calendar entry dialog. We can use the beginDialogAction method on a dialog to define triggers that start new dialogs on top of the AddCalendarEntry dialog.

bot.dialog('AddCalendarEntry', [

    ...

])

    .beginDialogAction('AddCalendarEntryHelp', 'AddCalendarEntryHelp', { matches: /^help$/ });

bot.dialog('AddCalendarEntryHelp', (session, args, next) => {

    let msg = "Add Calendar Entry Help: we need the time of the meeting, the subject and the location to create a new appointment for you.";

    session.endDialog(msg);

});

When we run this, we get the desired effect, as shown in Figure 5-41.

We will dive deeper into actions and their behavior in the following chapter.

Recognizers

Recall we defined that the help dialog will be triggered via a regular expression. How does the Bot Builder SDK implement this? This is where recognizers come in. A recognizer is a piece of code that accepts incoming messages and determines what the user’s intent was. A recognizer returns an intent name and a score. The intent and score can come from an NLU service like LUIS, but they do not have to.

By default, as seen in the previous examples, the only recognizer in our bot is a regular expression or plain-text matcher. It takes in a regular expression or a hard-coded string and matches it to the incoming message’s text. We could utilize an explicit version of this recognizer by adding a RegExpRecognizer to our bot’s list of recognizers. The following implementation states that if the user’s input matches the provided regular expression, an intent called HelpIntent is resolved with a score of 1.0. Otherwise, the score is 0.0.

bot.recognizer(new builder.RegExpRecognizer('HelpIntent', /^help$/i));

bot.dialog('help', (session, args, next) => {

    session.endDialog("Hi, I am a calendar concierge bot. I can help you make and cancel appointments!");

})

    .triggerAction({

        matches: 'HelpIntent',

        onSelectAction: (session, args, next) => {

            session.beginDialog(args.action, args);

        }

    });

Another thing that the recognizer model allows us to do is to create a custom recognizer that executes any code we want and resolves an intent with a score. Here’s an example:

bot.recognizer({

    recognize: (context, done) => {

        var intent = { score: 0.0 };

        if (context.message.text) {

            if (context.message.text.toLowerCase().startsWith('help')) intent = { score: 1.0, intent: 'HelpIntent' };

        }

        done(null, intent);

    }

});

Now this is quite a simple example, but our minds should be racing with the possibilities. For example, if the user’s input is nontext media such as an image or video, we can write a custom recognizer that validates the media and responds accordingly.

The Bot Builder SDK allows us to register multiple recognizers to our bot. Whenever a message comes into the bot, each recognizer is invoked, and the recognizer with the highest score is deemed the winner. If two or more recognizers result in the same score, the recognizer that was registered first wins.

Lastly, this same mechanism can be used to connect our bot to LUIS, and in fact the Bot Builder SDK includes a recognizer for this very case. To do this, we take the endpoint URL for our LUIS application (perhaps the one we created in Chapter 3) and use it as a parameter to the LuisRecognizer.

bot.recognizer(new builder.LuisRecognizer('https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/{APP_ID}?subscription-key={SUBSCRIPTION_KEY}}'));

Once we set this up, we add a triggerAction call for each intent we would like to handle globally, as we did with the help dialog. The strings passed as the “matches” member must correspond to our LUIS intent names.

bot.dialog('AddCalendarEntry', [

    ...

])

.beginDialogAction('AddCalendarEntryHelp', 'AddCalendarEntryHelp', { matches: /^help$/ })

.triggerAction({matches: 'AddCalendarEntry'});

bot.dialog('RemoveCalendarEntry', [

    ...

])

.triggerAction({matches: 'DeleteCalendarEntry'});

At this point, our bot conversation can navigate between the dialogs by using LUIS intents (Figure 5-42). LUIS’s intent and entity objects are passed into the dialogs.

Exercise 5-2

Connecting Your Bot to LUIS

In this task you will connect a bot to the LUIS application you created in Chapter 3.

1. 1.

   Create an empty bot and create a dialog to handle each type of intent created in Chapter 3. For each dialog, simply send a message with the dialog name.

    
2. 2.

   Register the LUIS recognizer with your bot and confirm that it works.

    
3. 3.

   The first method of every dialog waterfall is passed the session object and an args object. Use a debugger to explore the objects.⁵ What is the structure of data from LUIS? Alternatively, send the JSON string representing the args object to the user.

    

Recognizers are a powerful feature in the Bot Builder SDK , allowing us to equip our bots with a variety of behaviors based on incoming messages.

Building a Simple Calendar Bot

Ideally the patterns around how we structure a conversation are becoming clear. The git repos provided with the book include a Calendar Concierge Bot that we build on throughout the remaining chapters in the book. Every chapter that makes changes to the bot has its own folder in the repo. The Chapter 5 folder includes the skeleton code that integrates with LUIS and sends back a message saying what the bot understood. Auth and API integration will be covered in Chapter 7. We add basic multilanguage support in Chapter 10, human handover in Chapter 12, and analytics integration in Chapter 13.

Conclusion

This was quite an introduction into the Bot Framework and Bot Builder SDK. We are now equipped to build basic bot experiences. The core concepts of creating bot channel registrations, connecting our bots to channel connectors, debugging using the Bot Framework Emulator and ngrok, and building bots using the Bot Builder SDK are the key pieces we need to understand to be productive. The Bot Builder SDK is a powerful library to assist us in the process. We introduced the core concepts from the SDK. Without getting too deep into the details of the SDK, we developed a chat bot that can interpret a large variety of natural language inputs that execute the use cases we aimed to support from Chapter 3. The only thing left to do is to pull in a calendar API and translate the LUIS intent and entity combinations into the right API calls.

Before we jump into this, we will dive deeper into the Bot Builder SDK to make sure we are selecting the correct approach in our final implementation.

Conversation State

As mentioned throughout the previous chapters, a good conversational engine will store each user and conversation’s state so that whenever a user communicates with the bot, the right state of the conversation flow is retrieved, and there is a coherent experience for the user. In the Bot Builder SDK, this state is, by default, stored in memory via the aptly named MemoryBotStorage. Historically, state was stored in a cloud endpoint; however, this has been deprecated. Every so often, we may run into a reference to the state service in some older documentation, so be aware that it no longer exists.

In addition, as a dialog is executing, we have access to its state object referred to as dialogData . Any time a message is received from a user, the Bot Builder SDK will retrieve the user’s state from the state storage, populate the three data objects plus dialogData on the session object, and execute the logic for the current step in the conversation. Once all responses are sent out, the framework will save the state back into the state storage.

In some of the code from the previous chapter, there were instances where we had to re-create a custom object from dialogData and then store the object into the dialogData. The reason for this is that saving an object into the dialogData (or any of the other state containers) will turn the object into a vanilla JavaScript object, like using JSON.stringify would. Trying to invoke any method on session.dialogData.addEntry in the previous code, before resetting to a new object, would cause an error.

The storage mechanism is implemented by an interface called IBotStorage.

The ChatConnector class that we instantiate when building a new instance of a bot installs the default MemoryBotStorage instance, which is a great option for development. The SDK allows us to provide our own implementation to replace the default functionality, something you will most likely want to do in a production deployment as this ensures that states are stored instead of being erased any time your instances restarts. For instance, Microsoft provides two additional implementations of the interface, a NoSQL implementation for Azure Cosmos DB¹ and an implementation for Azure Table Storage.² Both are Azure services available through the Azure Portal. You can find the two storage implementations in the botbuilder-azure node package, documented at https://github.com/Microsoft/BotBuilder-Azure . You are also able to write your own IBotStorage implementation and to register it with the SDK. Writing your own implementation is a matter of following the simple IBotStorage interface.

Messages

In the previous chapter, our bot communicated to the user by sending text messages using either the session.send or session.endDialog method. This is fine, but it limits our bot a fair amount. A message between a bot and a user is composed of a variety of pieces of data that we ran into in the “Bot Builder SDK Basics” section in the previous chapter.

The Bot Builder IMessage interface defines what a message is really composed of.

For this chapter, we will be most interested in the text, attachments, suggestedActions, and attachmentLayout as they form the basis of a good conversational UX.

To create a message object in code, we create a builder.Message object. At that point, you can assign the properties as per the following example. A message can then be passed into the session.send method.

Likewise, when a message comes into your bot, the session object contains a message object. Same interface. Same type of data. But, this time, it is coming in from the channel rather than from the bot.

Note that IMessage inherits from IEvent, which means it has a type field. This field is set to message for an IMessage, but there are other events that may come from either the framework or a custom app.

We can register a handler for each event type by using the on method on the UniversalBot. The resulting conversation with a bot that handles events can provide for more immersive conversational experiences for your users (Figure 6-1).

const bot = new builder.UniversalBot(connector, [

    (session) => {

    }

]);

bot.on('conversationUpdate', (data) => {

    if (data.membersAdded && data.membersAdded.length > 0) {

        if (data.address.bot.id === data.membersAdded[0].id) return;

        const name = data.membersAdded[0].name;

        const msg = new builder.Message().address(data.address);

        msg.text('Welcome to the conversation ' + name + '!');

        msg.textLocale('en-US');

        bot.send(msg);

    }

});

bot.on('typing', (data) => {

    const msg = new builder.Message().address(data.address);

    msg.text('I see you typing... You\'ve got me hooked! Reel me in!');

    msg.textLocale('en-US');

    bot.send(msg);

});

Addresses and Proactive Messages

In the message interface, the address property uniquely represents a user in a conversation. It looks like this:

The importance behind an address is that we can use it to send a message proactively outside the scope of a dialog. For example, we could create a process that sends a message to a random address every five seconds. This message has zero effect on the user’s dialog stack.

If we did want to modify the dialog stack, perhaps by calling into a complex dialog operation, we can utilize the beginDialog method on the UniversalBot object.

The significance of these concepts that we can have external events in disparate systems begin affecting the state of a user’s conversation within the bot. We will see this applied in the context of OAuth web hooks in the next chapter.

Rich Content

Rich content can be sent to the user using the attachments functionality in the BotBuilder IMessage interface. In the Bot Builder SDK, an attachment is simply a name, content URL, and a MIME type.³ A message in the Bot Builder SDK accepts zero or more attachments. It is up to the bot connectors to translate that message into something that the channel will understand. All types of messages and attachments are not supported by every channel. Be careful when creating attachments of various MIME types.

For example, to share an image, we can use the following code:

Figure 6-2 shows the resulting user interface in the emulator, and Figure 6-3 shows it in Facebook Messenger. We could imagine similar rendering in other platforms.

This code will send audio file attachments, which can be played right from within the messaging channel.

Figure 6-4 shows the emulator, and Figure 6-5 shows Facebook Messenger ().

Whoops! It seems like OGG⁴ files are not supported. This is a good example of Bot Framework behavior when our bot sends an invalid message to Facebook or any other channel. We will investigate this further in the “Channel Errors” section later in this chapter. My console error log has this message:

If we look at the error list in the Bot Framework Messenger Channels page, we should find another clue like in Figure 6-6.

OK, so they make it somewhat easy to diagnose the problem. We know we must provide a different file format. Let’s try an MP3.

You can see the resulting Emulator and Facebook Messenger renderings in Figure 6-7 and Figure 6-8.

The Emulator still produces a link, but Messenger has a built-in audio player you can utilize! The experience uploading a video is similar. Messenger will provide a built-in video player right within the conversation.

Buttons

Bots can also send buttons to users. A button is a distinct call to action for a user to perform a task. Each button has a label associated with it, as well as a value. A button also has an action type, which will determine what the button does with the value when the button is clicked. The three most common types of actions are open URL, post back, and IM back. Open URL typically opens a web view within the messaging app or a new browser window in a desktop setting. Both post back and IM back send the value of the button as a message to the bot. The difference between the two is that clicking the post back should not display a message from the user in the chat history, whereas the IM back should. Not all channels implement both types of buttons.

Note that in the previous code we used a CardAction object. A CardAction is an encapsulation of the data we discussed earlier: a type of action, a title, and value. The channel connectors will usually render a CardAction into a button on the individual platforms.

Figure 6-9 shows what running this code looks like in the emulator, and Figure 6-10 shows it in Facebook Messenger. If we click the Open Google button in the emulator, it opens the web page in your default browser. We first click Im Back, and then once we receive the response card, we click Post Back. Note that Im Back sent a message and the message appears in the chat history, whereas the Post Back button sent a message that the bot responds to, but the message does not appear in the chat history.

Messenger works a bit differently.⁵ Let’s look at the mobile app behavior. If we click Open Google, a web view will show up that covers about 90 percent of the screen. If we click Im Back and Post Back, the app exhibits the same behavior. Messenger only supports post back; in addition, the message value is never showed to the user. The chat history contains only the title of the button that was clicked.

Of course, not all channels support all types. In addition, channels may natively support other functionality that the Bot Builder SDK is not. For example, Figure 6-11 shows the documentation for actions Messenger supports through its button templates as of the time of this writing. We will look at utilizing native channel functionality later in this chapter.

In the Bot Builder SDK, every card action can be created by using the static factory methods in the CardAction class. Here is the relevant code from the Bot Builder source:

Cards

Another type of Bot Builder attachment is the hero card. In our previous example with button actions, we conveniently ignored the fact that button actions need to be part of a hero card object, but what is that?

The term hero card originates from the racing world. The cards themselves are usually bigger than baseball cards and are designed to promote a race team, specifically the driver and sponsors. It would include photos, information about the driver and sponsors, contact information, and so on. But really the concept is reminiscent of typical baseball or Pokémon cards.

In the context of UX design, a card is an organized way of displaying images, text, and actions. Google brought cards to the masses when it introduced the world to its Material Design⁶ on Android and the Web. Figure 6-12 shows two examples of card design from Google’s Material Design documentation. Notice the distinct usage of images, titles, subtitles, and calls to action.

In the context of bots, the term hero card refers to a grouping of an image with text, buttons for actions, and an optional default tap behavior. Different channels will call cards different things. Facebook loosely refers to them as templates . Other platforms just refer to the idea as attaching content to a message. At the end of the day, the UX concepts are the same.

In the Bot Builder SDK, we can create a card using the following code. We also show how this card renders in the emulator (Figure 6-13) and on Facebook Messenger (Figure 6-14).

const bot = new builder.UniversalBot(connector, [

    (session) => {

        const cardActions = [

            builder.CardAction.openUrl(session, 'http://google.com', "Open Google"),

            builder.CardAction.imBack(session, "Hello!", "Im Back"),

            builder.CardAction.postBack(session, "Hello!", "Post Back")

        ];

        const card = new builder.HeroCard(session)

            .buttons(cardActions)

            .text('this is some text')

            .title('card title')

            .subtitle('card subtitle')

            .images([new builder.CardImage(session).url("https://bot-framework.azureedge.net/bot-icons-v1/bot-framework-default-7.png").toImage()])

            .tap(builder.CardAction.openUrl(session, "http://dev.botframework.com"));

        const msg = new builder.Message(session).text("sample actions").addAttachment(card);

        session.send(msg);

    }

]);

Cards are a great way to communicate the results of a bot action invoked by the user. If you would like to display some data with an image and follow-up actions, there is no better way to do so than using cards. The fact that you get only a few different text fields, with limited formatting abilities, means that the UX resulting in this approach can be a bit limited. That is by design. For more complex visualizations and scenarios, you can either utilize adaptive cards or render custom graphics. We will explore both topics in Chapter 11.

The next question is, can we display cards side by side in a carousel style? Of course, we can. A message in the Bot Builder SDK has a property called attachmentLayout . We set this to carousel, add more cards, and we’re done! The emulator (Figure 6-15) and Facebook Messenger (Figure 6-16) take care of laying the cards out in a friendly carousel format. The default attachmentLayout is a list. Using this layout, the cards would appear one below the other. It is not the most user-friendly approach.

const bot = new builder.UniversalBot(connector, [

    (session) => {

        const cardActions = [

            builder.CardAction.openUrl(session, 'http://google.com', "Open Google"),

            builder.CardAction.imBack(session, "Hello!", "Im Back"),

            builder.CardAction.postBack(session, "Hello!", "Post Back")

        ];

        const msg = new builder.Message(session).text("sample actions");

        for(let i=0;i<3;i++) {

            const card = new builder.HeroCard(session)

                .buttons(cardActions)

                .text('this is some text')

                .title('card title')

                .subtitle('card subtitle')

                .images([new builder.CardImage(session).url("https://bot-framework.azureedge.net/bot-icons-v1/bot-framework-default-7.png").toImage()])

                .tap(builder.CardAction.openUrl(session, "http://dev.botframework.com"));

            msg.addAttachment(card);

        }

        msg.attachmentLayout(builder.AttachmentLayout.carousel);

        session.send(msg);

    }

]);

Cards can be a bit tricky because there are many ways of laying out buttons and images. Each platform has ever so slightly different rules. On some platforms, openUrl buttons (but not others) must point to an HTTPS address. There may also be rules that limit the number of buttons per card, number of cards in a carousel and image aspect ratios. Microsoft’s Bot Framework will handle all this in the best way it can, but being aware of these limitations will help us debug our bots.

Suggested Actions

We’ve discussed suggested actions in the context of conversational design; they are message-context-specific actions that can be performed immediately after a message is received. If another message comes in, the context is lost, and the suggested actions disappear. This is in opposition to card actions, which stay on the card in the chat history pretty much forever. The typical UX for suggested actions, also referred to as quick replies , is as a horizontally laid out list of buttons along the bottom of the screen.

The code for building suggested actions is similar to a hero card, except the only data we need is a collection of CardActions. The type of actions allowed in the suggested actions area will depend on the channel. Figure 6-17 and Figure 6-18 shows renderings on the emulator and Facebook Messenger, respectively.

msg.suggestedActions(new builder.SuggestedActions(session).actions([

    builder.CardAction.postBack(session, "Option 1", "Option 1"),

    builder.CardAction.postBack(session, "Option 2", "Option 2"),

    builder.CardAction.postBack(session, "Option 3", "Option 3")

]));

The suggested actions buttons are great to keep the conversation with the user going without asking the user to guess what they can type into the text message field.

Channel Errors

In the “Rich Content” section, we noted that when a bad request is sent by our bot to the Facebook Messenger connector, our bot will receive an HTTP error. This error was also printed out in the console output of the bot. It seems that the Facebook Bot connector is reporting an error from the Facebook APIs back to our bot. That is cool. The additional feature we saw was that the channel detail page in Azure also contained all those errors. Although minor, this is a powerful feature. It allows us to quickly see how many messages were rejected by the API and the error codes. The case we ran into, that a specific file type format was not supported, was just one of many possible errors. We would see errors if the message is malformed, if there are authentication issues, or if Facebook rejects the connector message for any other reason. Similar ideas apply to the other set of connectors. In general, the connectors are good at translating Bot Framework activities into something that will not be rejected by the channels, but it happens.

In general, if our bot sends a message to a Bot Framework connector and the message does not appear on the interface, chances are there was an issue with the interaction between the connector and channel, and this online error log will contain information about the failure.

Channel Data

We have mentioned several times that different channels may render messages differently or have different rules about certain items, such as the number of hero cards in a carousel or the number of buttons in a hero card. We have been showing examples of Messenger and emulator renderings, as those channels typically work well. Skype is another one that supports a lot of the Bot Builder features (which makes sense, as both are owned by Microsoft). Slack does not have as much rich support for these features, but its editable messages are a slick feature we will visit in Chapter 8.

For illustration purposes, Figure 6-19 is what the carousel with the suggested actions discussed earlier looks like in Slack.

That’s not a carousel. There is no such concept in Slack! There are also no cards to speak of; it is just messages with attachments. The images are not clickable either; the default link is displayed above the image. Both the Im Back and Post Back buttons appear to do a post back. There is no concept of suggested actions/quick replies. You can find more information about the Slack Message format online.⁷

However, the team behind the Bot Builder SDK has thought of the issue where you may want to specify the exact native channel message, distinct from the default Bot Framework connector rendering for that channel. The solution is to provide a field on the Message object that contains the native channel JSON data for incoming messages and a field that may contain native channel JSON responses.

The terminology used in the Node SDK is sourceEvent (the C# version of Bot Builder refers to this concept as channelData). The sourceEvent in the Node SDK exists on the IEvent interface. Remember, this is the interface that IMessage implements as well. This means any event from a bot connector may include the raw channel JSON.

Let’s look at a feature in Facebook Messenger that is not readily supported by the Bot Framework. By default, cards in Messenger require an image with a 1.91:1 aspect ratio.⁸ The default conversion of a hero card by the connector utilizes this template. There is, however, the ability to utilize a 1:1 image ratio. There are other options in the documentation that are hidden by the Bot Framework. For example, Facebook has a specific flag around setting cards as sharable. Furthermore, you can control the size of the WebView invoked by an openURL button in Messenger. For now, we will stick to modifying the image aspect ratio.

For starters, let’s see the code to send the same card we have been sending using the hero card object but using Facebook’s native format:

The rendering (Figure 6-20) looks identical to the rendering using the hero card.

We set image_aspect_ratio to square, and now Facebook renders it as a square (Figure 6-21)!

const msg = new builder.Message(session);

msg.sourceEvent({

    facebook: {

        attachment: {

            type: 'template',

            payload: {

                template_type: 'generic',

                image_aspect_ratio: 'square',

                // more...

            }

        }

    }

});

session.send(msg);

It’s that easy! This is just a taste. In Chapter 8, we will explore using the Bot Framework to integrate with native Slack features.

Group Chat

Some types of bots are meant to be used in a group setting. In the context of Messenger, Twitter direct messages, or similar platforms, the interaction between a user and a bot is typically one on one. However, some channels, most notably Slack, are focused on collaboration. In such a context, the ability to converse with multiple users simultaneously becomes important. Giving your bot the ability to productively participate in a group conversation as well as to handle mention tags correctly is important.

Some channels will allow the bot to view every single message that is sent between users in a channel. Other channels will only send messages to the bot if it is mentioned (for example, “hey @szymonbot, write a book on bots will ya?”).

If we are in a channel that allows our bot to see all messages in a group setting, our bot could monitor the conversation and silently execute code based on the discussion (because replying to every message on a group conversation is kind of annoying), or it could ignore everything that doesn’t have a mention of the bot. It could also implement a combination of the two behaviors, where the bot is activated by a mention with a certain command and becomes chatty.

In the “Messages” section, we showed the interface for a message. We glossed over the entities list, but it becomes relevant here. One type of entity we may receive from a connector is mentions. The object includes the name and id of the mentioned user and looks as follows:

Facebook does not support this type of entity, but Slack does. We will connect a bot to Slack in Chapter 8, but in the meantime, here is the code that could always reply in a direct messaging scenario but only reply in a group chat if it is mentioned:

Figure 6-22 is what the experience looks like in Slack in a direct conversation.

Figure 6-23 shows the behavior in a group chat (excuse the overly original username srozga2).

Custom Dialogs

We have constructed our dialogs by using the bot.dialog(…) method. We also discussed the concept of a waterfall. In the calendar bot we started in the previous chapter, each of our dialogs was implemented via waterfalls: a set of steps that will execute in sequence. We can skip some steps or end the dialog before all steps are completed, but the idea of a predefined sequence is key. This logic is implemented by a class in the Bot Builder SDK called WaterfallDialog. If we look at the code behind the dialog(…) call, we will find this bit:

What if the conversation piece we would like to encode is not easily represented in a waterfall abstraction? What choices do we have? We can create a custom implementation of a dialog!

In the Bot Builder SDK, a dialog is a class that represents some interaction between the user and the bot. Dialogs can call other dialogs and accept return values from those child dialogs. They live on a dialog stack, not unlike a function call stack. Using the default waterfall helper hides some of these details; implementing a custom dialog brings us closer to the dialog stack reality. The abstract Dialog class from the Bot Builder is shown here:

To illustrate the concepts, we create a BasicCustomDialog. Since Bot Builder is written in TypeScript,⁹ a typed superset of JavaScript, we went ahead and wrote the subclass in TypeScript, compiled into JavaScript using the TypeScript Compiler (tsc), and then used it in app.js.

Let’s look at the custom dialog’s code. This happens to be TypeScript as it has a cleaner interface when using inheritance; the compiled JavaScript is shown later. When the dialog begins, it send the “begin” text. When it receives a message, it responds with the “reply received” text. If the user sent the “prompt” text, the dialog will ask the user for some text input. It would then receive the text input in the dialogResumed method, which prints that result. If the user had entered “done,” the dialog finishes and returns to the root dialog.

We use an instance of the dialog directly in app.js. In the default waterfall, we echo any message, except the “custom” input, which begins the custom dialog.

Figure 6-24 shows what a sample interaction looks like.

Incidentally, the Promps.text, Prompts.number, and other Prompt dialogs are all implemented as custom dialogs.

The compiled JavaScript for the custom dialog is shown next. It is a bit more challenging to reason about, but at the end of the day, it is standard ES5 JavaScript prototype inheritance.¹⁰

Actions

We now have a good idea of how powerful abstraction dialogs are and how the Bot Builder SDK manages the dialog stack. One of the key pieces of the framework that we do not have good insight into is how to link user actions to transformations of the dialog stack. At the most basic level, we can write code that simply calls beginDialog. But how do we make that determination based on user input? How can we hook that into the recognizers that we learned about in the previous chapter and specifically LUIS? That is what actions allow us to do.

The Bot Builder SDK contains six types of actions, with two being global and four scoped to a dialog. The two global actions are triggerAction and customAction. We’ve run into triggerAction before. It allows the bot to invoke a dialog when an intent is matched at any point during the conversation, assuming the intent does not match a dialog-scoped action beforehand. These are evaluated any time user input is received. The default behavior is to clear the entire dialog stack before the dialog is invoked.

Each of our main dialogs in our code in the calendar bot from the previous chapter uses the default triggerAction behavior, except for Help. The Help dialog is invoked on top of the dialog stack, so when it completes, we are back to whatever dialog the user was on to begin on. To achieve this effect, we override the onSelectAction method and specify the behavior we want.

A customAction binds directly to the bot object, instead of a dialog. It allows us to bind a function to respond to user input. We don’t get a chance to query the user for more information like a dialog implementation would. This is good for functionality that simply returns a message or performs some HTTP call based on user input. In fact, we could as far as to rewrite the Help dialog like this. The code looks straightforward, but we lose the encapsulation and extensibility of the dialog model. In other words, we no longer have the logic in its own dialog, with the ability to execute several steps, collect user input, or provide a result to the calling object.

The four types of contextual actions are beginDialogAction, reloadAction, cancelAction, and endConversationAction. Let’s examine each one.

BeginDialogAction creates an action that pushes a new dialog on the stack whenever the action is matched. Our contextual help dialogs in the calendar bot used this approach. We created two dialogs: one as the help for the AddCalendarEntry dialog and the second as a help for the RemoveCalendarEntry dialog.

Our AddCalendarEntry dialog can then bind the beginDialogAction to its appropriate help dialog.

Note that the behavior of this action is the same as calling beginDialog manually. The new dialog is placed on top of the dialog stack, and the current dialog is continued when done.

The reloadAction call performs a replaceDialog. replaceDialog is a method on the session object that ends the current dialog and replaces it with an instance of a different dialog. The parent dialog does not get a result until the new dialog finishes. In practice, we can utilize this to restart an interaction or to switch into a more appropriate dialog in the middle of a flow.

Here is the code for the conversation (see Figure 6-25):

lib.dialog(constants.dialogNames.AddCalendarEntry, [

    // code

])

    .beginDialogAction(constants.dialogNames.AddCalendarEntryHelp, constants.dialogNames.AddCalendarEntryHelp, { matches: constants.intentNames.Help })

    .reloadAction('startOver', "Ok, let's start over...", { matches: /^restart$/i })

    .triggerAction({ matches: constants.intentNames.AddCalendarEntry });

CancelAction allows us to cancel the current dialog. The parent dialog will receive a cancelled flag set to true in its resume handler. This allows the dialog to properly act on the cancellation. The code follows (the conversation visualization is shown in Figure 6-26):

lib.dialog(constants.dialogNames.AddCalendarEntry, [

    // code

])

    .beginDialogAction(constants.dialogNames.AddCalendarEntryHelp, constants.dialogNames.AddCalendarEntryHelp, { matches: constants.intentNames.Help })

    .reloadAction('startOver', "Ok, let's start over...", { matches: /^restart$/i })

    .cancelAction('cancel', 'Cancelled.', { matches: /^cancel$/i})

    .triggerAction({ matches: constants.intentNames.AddCalendarEntry });

Lastly, the endConversationAction allows us to bind to the session.endConversation call. Ending a conversation implies that the entire dialog stack is cleared and that all the user and conversation data is removed from the state store. If a user starts messaging the bot again, a new conversation is created without any knowledge of the previous interactions. The code is as follows (Figure 6-27 shows the conversation visualization):

lib.dialog(constants.dialogNames.AddCalendarEntry, [

    // code

])

    .beginDialogAction(constants.dialogNames.AddCalendarEntryHelp, constants.dialogNames.AddCalendarEntryHelp, { matches: constants.intentNames.Help })

    .reloadAction('startOver', "Ok, let's start over...", { matches: /^restart$/i })

    .cancelAction('cancel', 'Cancelled.', { matches: /^cancel$/i})

    .endConversationAction('end', "conversation over!", { matches: /^end!$/i })

    .triggerAction({ matches: constants.intentNames.AddCalendarEntry });

Libraries

Libraries are a way of packaging and distributing related bot dialogs, recognizers, and other functionality. Libraries can reference other libraries, resulting in bots with highly composed pieces of functionality. From the developer perspective, a library is simply a nicely packaged collection of dialogs, recognizers, and other Bot Builder objects with a name and, commonly, a set of helper methods to aid in invoking the dialogs and other library-specific features. In our Calendar Concierge Bot in Chapter 5, each dialog was part of a library related to a high-level bot feature. The app.js code loads all the modules and then installs them into the main bot via the bot.library call.

This is library composition in action: UniversalBot is itself a subclass of Library. Our main UniversalBot library imports six other libraries. A reference to a dialog from any other context must be namespaced using the library name as a prefix. From the perspective of the root library or dialogs in the UniversalBot object, invoking any other library’s dialog must use a qualified name in the format: libName:dialogName. This fully qualified dialog name referencing process is necessary only when crossing library boundaries. Within the context of the same library, the library prefix is not necessary.

A common pattern is to expose a helper method in your module that invokes library dialog. Think of it as library encapsulation; a library should not know anything about the internals of another library. For example, our help library exposes a method to do just that.

Conclusion

Microsoft’s Bot Builder SDK is a powerful bot construction library and conversation engine that helps us develop all types of asynchronous conversational experiences from simple back and forth to complex bots with a multitude of behaviors. The dialog abstraction is a powerful way of modeling a conversation. Recognizers define the mechanisms that our bot utilizes to translate user input into machine-readable intents. Actions map those recognizer results into operations on the dialog stack. A dialog is principally concerned with three things: what happens when it begins, what happens when a user’s message is received, and what happens when a child dialog returns its result. Every dialog utilizes the bot context, called the session, to retrieve the user message and to create responses. A response may be composed of text, video, audio, or images. In addition, cards can produce richer and context-sensitive experiences. Suggested actions are responsible for keeping the user from guessing what to do next.

In the following chapter, we’ll apply these concepts to integrate our bot with the Google Calendar API, and we’ll take steps to creating a compelling first version of our calendar bot experience.

A Word on OAuth 2.0

This isn’t a book about security, but understanding basic authentication and authorization mechanisms is essential to be a developer. OAuth 2.0 is a standard authorization protocol. The three-legged OAuth 2.0 flow allows third-party applications to access services on behalf of another entity. In our case, we will be accessing a user’s Google Calendar data on behalf of that user. At the end of the three-legged OAuth flow, we end up with two tokens: an access token and a refresh token. The access token is included in requests to an API in the authorization HTTP header and provides data to the API declaring which user we are requesting data. Access tokens are typically short-lived to reduce the window during which a compromised access token can be utilized. When an access token expires, we can use the refresh token to receive a new access token.

To initiate the flow, we first redirect the user to a service that they can authenticate against, say, Google. Google presents an OAuth 2.0 login page where it authenticates the user and asks the user for their consent so that the bot can access the user’s data from Google on their behalf. When authentication and consent are successful, Google sends an authorization code back into the bot’s API, via what’s known as the redirect URI. Finally, our bot requests the access and refresh tokens by presenting the authorization code to Google’s token endpoint. Google’s OAuth libraries will help us implement the three-legged flow in our calendar bot.

Setting Up Google APIs

Before we jump into it, we should set ourselves up to be able to use the Google APIs. Luckily, Google makes this quite easy via the Google Cloud Platform API console. Google Cloud Platform is Google’s Azure or AWS; it is Google’s one-stop shop for provisioning and managing different cloud services. To get started, we navigate to https://console.cloud.google.com . If this is our first time visiting the site, we will be asked to accept the terms of service. After that, we will be placed in the dashboard (Figure 7-1).

Our next steps are as follows. We will create a new project. Within that project, we will ask for access to the Calendar API. We will also give our project the ability to log in on behalf of users using OAuth2. Once done, we will receive a client ID and secret. Those two pieces of data, plus our redirect URI, are sufficient for us to use the Google API libraries within our bot.

Click the Select a Project drop-down. You’ll be met with a pop-up that, if you have not used this console before, should be empty (Figure 7-2).

Click the + button to add a new project. Give the project a name. Once the project is created, we will be able to navigate to it through the Select a Project functionality (Figure 7-3). The project is also assigned an ID, prefixed by the project name.

When the open the project, we see the project dashboard, which initially looks intimidating (Figure 7-4). There are many things we can do here.

Let’s begin by getting access to the Google Calendar API. We first click APIs & Services. We can find this link in the first few items on the left navigation pane. The page already has quite a few things populated. These are the default Google Cloud Platform services. Since we’re not using them, we can disable each one. Once ready, we can click the Enable APIs and Services button. We search for Calendar and click Google Calendar API. Finally, we click the Enable button to add it to our project (Figure 7-5). We will receive a warning indicating that we may need credentials to use the API. No problem, we will do this next.

To set up authorization, we click the Credentials link on the left pane. We will be met with a prompt to create credentials. For our use case, in which we will be accessing the user’s calendar, we need an OAuth Client ID¹ (Figure 7-6).

We will first be asked to set up the consent screen (Figure 7-7). This is the screen that the user will be shown when authenticating against Google. Most of us have probably encountered these types of screens across different web applications. For example, whenever we log into an app via Facebook, we will be presented with a page telling us that the app needs permission to read all your contact information and photos and even deepest secrets. This is Google’s way of setting up a similar page. It asks for data such as the product name, logos, terms of service, privacy policy URLs, and so on. To test the functionality we minimally need a product name.

At this point, we will be taken back to the Create Client ID function. As the Application Type setting, we should select Web Application and give our client a name and a redirect URI (Figure 7-8). We utilize our ngrok proxy URI (see Chapter 5 for more on ngrok). For local testing, we are free to enter a localhost address. For example, you can enter http://localhost:3978.

Once we click the Create button , we will receive a pop-up with the client ID and client secret (Figure 7-9). Copy them because we will need the values in our bot. If we lose the client ID and secret, we can always access them by navigating to the Credentials page for the project and selecting the entry we created in the OAuth 2.0 Client IDs.

At this point we are ready to hook our bot up to the Google OAuth2 provider.

Integrating Authentication with Bot Builder

We will need to install the googleapis node package as well as crypto-js, a library that lets us encrypt data. When we send the user to the OAuth login page, we also include a state in the URL. A state is simply a payload that our application can use to identify a user and their conversation. When Google sends back an authorization code as part of the OAuth 2.0 three-legged flow, it will also send back the state. The state parameter should be something recognizable to our API but very hard for a malicious actor to guess, such as a session hash or some other information we are interested in. Once we receive it from Google’s auth page, we can continue the user’s conversation using the data in the state parameter.

To mask the data from bad actors, we will encode this object as a Base64 string. Base64 is an ASCII representation of binary data.² Since a malicious actor could easily compromise this information by simply decoding from Base64, we will use crypto-js to encrypt the state string.

First, let’s install the two packages.

Second, let’s add three variables to our .env file representing the client ID, secret, and redirect URI. We use the redirect URI we provided in Figure 7-8 and the client ID and secret we received in Figure 7-9.

Third, we need to generate the URL to the login page and send a button that can open this URL. The Google Auth APIs can do a lot of this for us. We will do a few things in our code. First, we import the crypto-js and googleapis packages. Next, we create an OAuth2 client instance including our client data. The state that we will send as part of the login URL contains the user’s address. As shown in the previous chapter, an address is sufficient to uniquely identify a user’s conversation, and Bot Builder contains the facilities to help us send messages to that user by simply presenting the conversation address. We use crypto-js to encrypt the state, using the ASE algorithm.³ AES is a symmetric-key algorithm, which means that the data is encrypted and decrypted using the same key or passphrase. We add the passphrase into our .env file with the name AES_PASSPHRASE.

Another thing to note is the scopes array. When requesting authorization to the Google APIs, we specify to Google which APIs we are looking for access to using scopes. We can think of each item in the scopes array as a piece of data we want to access about the user from Google’s APIs. Of course, this array needs to be a subset of the APIs our Google project may access to begin with. If we added a scope we did not enable for our project earlier, the authorization process would fail.

We also need to be able to send a button for the user to utilize to authorize the bot. We utilize the built-in SigninCard for this purpose.

The emulator renders the SigninCard as per Figure 7-10.

At this point we can click the Login button to log into Google and authorize our bot to access our data, but it would fail because we have not yet provided the code to handle the message from the return URI. We use the same approach to install a handler for the https://a4b5518e.ngrok.io/oauth2callback endpoint as we did to install the API messages endpoint. We also enable restify.queryParser, which will expose each parameter in the query string as a field in the req.query object. For example, a callback in the form redirectUri?state=state&code=code will result in a query object with two properties, state and code.

We read the authorization code from the callback and use the Google OAuth2 client to get the tokens from the token endpoint. The tokens JSON will look like the following data. Note that expiry_date is the datetime in milliseconds since the epoch.⁴

Once we receive the tokens, we call setCredentials on the OAuth2 object, and it can now be used to access the Google Calendar API!

In the code location where we have access to the Calendar API, we can write code that gets a list of calendars that we own and prints out their names. Note that calapi in the following code is a helper object that wraps the Google Calendar API in JavaScript promises. The code is available in the chapter’s code library.

This code results in the following console output, which is an unfortunate reminder of the rather lonely workout schedule that has not seen much action since I became a dad.

Fatherhood weight gain aside, this is great! We do have a few challenges. We need to store the users’ OAuth tokens so we can access them any time the users message us. Where do we store them? This one is easy: private conversation data. How do we get access to that data dictionary in this context? We do this by passing the user’s address to the bot.loadSession method.

Recall that we stored in the user’s address into the encrypted state variable. We can decrypt that object by using the same passphrase we used to encrypt the data.

After we receive the token, we can load the bot session from the address. At that point, we have a session object that has all the dialog methods such as beginDialog for us to use.

The processUserCalendars dialog could look something like this. It sets the tokens into the private conversation data, lets the user know they are logged in, and displays the names of all of the client’s calendars.

The interaction would look like Figure 7-11.

Seamless Login Flow

We have successfully logged in and stored the access token, but we have not yet demonstrated a seamless mechanism to redirect to a login flow when a dialog requires our users to be logged in. More specifically, if in the context of the calendar bot a user is not logged in and asks the bot to add a new calendar entry, the bot should show the login button and then continue with the Add Entry dialog once login is successful.

We will naturally implement a Login dialog and a Logout dialog. Logout simply checks the existence of tokens in the conversation state. If we do not have the tokens, we are already logged out. If we do, we use Google’s library to revoke the user’s credentials.⁵ The tokens are no longer valid.

Login is a waterfall dialog that begins an EnsureCredentials dialog before it gets to the next step. In the second step, it verifies whether it is logged in. See the following code. It does this by verifying that it receives the authenticated flag from the EnsureCredentials dialog. If yes, it simply lets the user know that she is logged in. Otherwise, an error is shown to the user.

Notice what we did here. We outsourced the logic of figuring out if we are logged in, logging in, and then sending the result back to a different dialog. As long as that dialog returns with an object with fields authenticated and, optionally, error, this just works. We will use the same technique to inject an authorization flow into any other dialog that requires it.

We illustrate the flow for the first case in Figure 7-12. A dialog requests that we have the user’s authorizations before continuing, like the Login dialog did in the previous code. The user is sent to the auth page. Once the auth page returns a successful authorization code, it sends a callback to our oauth2callback. Once we get the tokens, we call a StoreTokens dialog to store the tokens into the conversation data. That dialog will return a success message to EnsureCredentials . In turn, this returns a successful authentication message to the calling dialog.

If an error occurs, the flow is similar except that we replace the EnsureCredentials dialog with the Error dialog. The Error dialog will then return a failed authenticate message to the calling dialog, which can handle the error as it best sees fit (Figure 7-13). Recall, as we noted in Chapter 5, replaceDialog is a call that replaces the current dialog on top of the stack with an instance of another dialog. The calling dialog does not know, nor care, about this implementation detail.

In the case that the user clicks the login button when a dialog is not expecting a reply and EnsureCredentials is not on top of the stack, the flow is slightly different. We want to display a success or failure message to the user if the authorization succeeds or fails. To achieve this, we will put a confirmation dialog, AuthConfirmation, on the stack before invoking the StoreTokens dialog (Figure 7-14).

Likewise, in the case we receive an authorization error, we push the AuthConfirmation dialog on top of the stack, before pushing the Error dialog (Figure 7-15). This will ensure that the confirmation dialog displays the right type of message to the user.

Let’s see what the code for this looks like. The Login and Logout dialogs are done, but let’s look at EnsureCredentials, StoreTokens, and Error.

EnsureCredentials is composed of two steps. First, if the user has a set of tokens defined, the dialog finishes passing a result indicating that the user is good to go. Otherwise, we create the auth URL and send a SigninCard to the user, just like we did in the previous section. The second step also executes in case 1. It simply tells the calling dialog that the user is authorized.

StoreTokens and Error are similar. Both essentially return an authorization result to its parent dialog. In the case of StoreTokens , we also store the tokens into the conversation data.

Note that EnsureCredentials is going to consume the result of either of these two and simply pass it down to the calling dialog. It is up to the calling dialog to display a success or error message. There may not even be a success message; the calling dialog may just jump into its own steps.

That covers cases 1 and 2. To ensure cases 3 and 4 are covered, we need to implement this AuthConfirmation dialog. The role of this dialog is to display either a success or failure message. Recall that we place either an Error (case 3) or StoreTokens (case 4) dialog on top of AuthConfirmation. The idea is that AuthConfirmation will receive the name of the dialog to place on top of itself and then send the appropriate message to the user when it receives a result.

Lastly, how do we change our endpoint callback code? Before we get there, we write a few helpers to invoke the different dialogs. We expose a function called isInEnsure that verifies whether we are getting into this piece of code from the EnsureCredentials dialog. This will dictate whether we need the AuthConfirmation. beginErrorDialog and beginStoreTokensAndResume both utilize this approach. Finally, ensureLoggedIn is the function that each dialog that requires authorization must call to kick off the flow.

Finally, let’s look at the callback. The code looks similar to our callback in the previous section, except we need to add the logic to begin the right dialogs. If we encounter any errors while loading our session object or we get an OAuth error, such as the user declining access to our bot, we redirect the user to the Error dialog. Otherwise, we use the authorization code from Google to get the tokens, set the credentials in the OAuth client, and call into the StoreTokens or AuthConfirmation dialog. The following code covers the four cases highlighted at the beginning of this section:

Integrating with the Google Calendar API

We are now ready to integrate with the Google Calendar API. There are a few things we should address first. Google Calendar allows users to have access to multiple calendars and, further, to have a different permission level for each calendar. In our bot, we assume that at any point we are querying or adding events into only one calendar, as flawed as that may seem. We could extend the LUIS application and bot to include the ability to specify a calendar for each utterance.

To handle this, we create a PrimaryCalendar dialog that allows users to set, reset, and retrieve their primary calendar. Similar to the EnsureCredentials dialog being called at the beginning of each dialog that requires authentication, we create a similar mechanism to guarantee that a calendar is set as primary.

Before we get there, let’s talk about connecting to the Google Calendar API. The googleapis node package includes the Calendar APIs, among others. The API utilizes the following format:

A calendar call would look as follows:

First, we will adapt this to the JavaScript Promise⁶ pattern. Promises make it easy to work with asynchronous calls. A promise in JavaScript represents an eventual completion or failure of an operation, as well as its return value. It supports a then method that allows us to perform an action on the result and a catch method that allows us to perform an action on the error object. Promises can be chained: the result of a promise can be passed to another promise that produces a result that can get passed into another promise and so forth, resulting in code that look as follows:

Our modified Google Calendar Promise API will look as follows:

We wrap all the necessary functions in a module called calendar-api. Some of the code is presented here:

The PrimaryCalendar dialog is a waterfall dialog with three steps. Step 1 ensures that the user is logged in by calling EnsureCredentials. Step 2 expects to receive a command from a user. We can either get our current primary calendar, set a calendar, or reset our calendar; thus, the three commands are get, set, or reset. Set calendar takes an optional calendar ID. If a calendar ID is not passed, the set command is treated equivalently to reset. Reset simply sends the user a list of all available calendars to which a user has write access to (another simplifying assumption).

The get case is handled by this code:

The reset case sends the user a carousel of calendar cards. If the user enters a text input, the third step of the waterfall assumes that the input is a calendar name and sets the right calendar. If the input isn’t recognized, an error message is sent.