Classifying Intents

We left off within the Build section. In the left pane, we have selected the Intents item. There is one intent in the system: None. This intent is resolved whenever the user’s input does not match any of the other intents. We could use this in our bot to tell the user that they are trying to ask questions outside of the bot’s area of expertise and remind them what the bot is capable of.

A typical workflow for using LUIS is to add an intent and present LUIS with several sample utterances that represent the intent. This is exactly what we will do. Figure 3-3 illustrates the process of creating an intent. The UI allows us to enter the utterance in a free-text entry field. We enter a sample, press Enter, enter another sample, press Enter, and so forth. Once we add enough sample utterances, we click the Save button and we're done with the intent (Figure 3-4).

Note that the user interface allows us to search for utterances, delete utterances, reassign intents to utterances, and display the data in a few different formats. Feel free to explore this functionality as you go along.

Before we add the rest of the intents, let’s see if we can train and test the application so far. Note that the Train button in the top right has a red indicator; this means the app has changes that have not yet been trained. Go ahead and click the Train button. Your request will be sent to the LUIS servers, and your app will be queued for training. You may notice a message that comes up informs you that LUIS is training your app and “0/2 completed.” The 2 is the number of classifier models that your application currently contains. One is for the None intent, and one is for AddCalendarEntry. When training is done, the Train button indicator will turn green to indicate that the app is up-to-date.

The intent interface also gives us information about which intent the latest trained app scores highest for each utterance (Figure 3-5). This piece of data is important because we can easily see when an application is trained to classify an utterance as one intent but assigns the highest score to a different intent. The discrepancy in the trained versus resulting intent is often an indicator that there is something in one or more models that is influencing the result in the wrong direction. We’ll cover this and other scenarios in the Troubleshooting section of this chapter. For now, it appears all our utterances have been successfully trained to result in a score of 1 on the AddCalendarEntry intent and between 0.05 and 0.07 on the None intent (see Figure 3-6); these numbers may vary depending on your exact utterances and also changes made by the LUIS engineering team.

Once trained, we can use the Test slide-out next to the Train button to test the models and see how they respond to different inputs (Figure 3-7). The Batch testing panel link allows a higher volume of testing to be performed. For our purposes, we will stick to the interactive mode.

The way LUIS functions is that it runs each input through all the models that were trained in the Training phase for our app. For each model, we receive a resulting score between 0 to 1 inclusive. The top-scoring intent is displayed prominently. Note that a score does not correspond to a probability. A score is dependent on the algorithm that is being used and usually represents some measure of the distance between the input to an intent’s ideal form. If LUIS scores an input with similar scores for more than one intent, we probably have some additional training to do.

After training our app and testing it, it seems to perform well until we try to break it. Then, it quickly starts looking wrong. Figure 3-8 illustrates this point.

Yikes. This is not terribly surprising. We have trained one intent with a limited number of utterances. We provided zero sample utterances to the None intent. This is the kind of behavior an undertrained model will exhibit. Let’s add some of these silly phrases to the None intent, train, and test again. You may try to add a few more nonsensical test cases like those in Figure 3-9. It should work better. We will not solve for all kinds of issues like this right now. This will take some time, dedication, and user feedback. But we should be aware that training the app what it should not know is as important as training an app what it should know.

Next, we will add the remaining intents. Figure 3-10, Figure 3-11, Figure 3-12, and Figure 3-13 show some sample utterances for the CheckAvailability, EditCalendarEntry, DeleteCalendarEntry, and ShowCalendarSummary intents.

Once all the intents are created and populated with sample utterances, we train and confirm that the predicted intents look accurate. You may note that although the top-scoring intent for each of the utterances is correct, the scores are rather low (Figure 3-14). This is an opportunity for us to train the app further. In fact, we can never assume that we can train an intent to be recognized with such a limited vocabulary and set of data. Getting NLU right requires patience, dedication, and thought. We will add more utterances to our app in an upcoming exercise.

Publishing Your Application

Obviously, we are not yet done developing our app. There are quite a few things missing and many details of LUIS we have not yet explored. We haven’t seen any real user data yet either. But, we can develop both the LUIS app and the consuming application in parallel. The process of taking our trained app and making it accessible via HTTP is referred to as publishing our app.

On the top navigation strip for the app, next to the Build section, we can find the Publish section. When we click this, we are greeted with a page that allows us to deploy the LUIS application (Figure 3-15). LUIS allows us to publish the application in one of two deployment slots: staging or production. Staging is meant for usage when we are still developing and testing the LUIS app. The production slot is meant to be used by production apps. The idea behind the two slots is that you can have a previous stable version of the LUIS app deployed into production, while you work on new app features in the staging slot.

We will go ahead and select the Staging slot from the “Publish to” drop-down. Once it’s published, we can access the app via an HTTP endpoint.

Before we test the resulting endpoint using cURL, a command line tool to transfer data over HTTP (among many other protocols), you may have noticed that below the publish settings there is an Add Key button and a set of keys for several deployment regions. When accessing a LUIS app, we must provide a key, which is how LUIS can bill us for API usage. LUIS is deployed to several regions; a key must be associated with a region. Keys are created using Microsoft’s Azure Portal. Azure is Microsoft’s cloud services umbrella. We will utilize it to register and deploy a bot in Chapter 5. To associate a key with an app, we must use the Add Key button. Lucky for us, LUIS provides a free starter key to use against apps published in the Staging slot.

Once we publish to the Staging slot, a few things happened. We now have information about the app version and the last time it was published. The URL under Starter_Key is now functional. We may enable verbose results (something we will examine momentarily) or Bing spell check integration (which we will discuss later in this chapter) via URL query parameters. Let’s take a closer look at the URL.

The first line of the URL is the service endpoint for the Azure Cognitive Services in the West US region and, specifically, our LUIS app. These are the query parameters that follow:

We can play with the API by making requests and seeing the responses by using curl. At its core, curl is a command-line tool to transfer data over a variety of protocols. We are going to use it to transfer data over HTTPS. You can find more information at https://curl.haxx.se/ . The command we can utilize is as follows. Note that we pass the subscription key as an HTTP header.

This query results in the following JSON. It gives us the score for each intent in our LUIS app.

You may be thinking, whoa, we just learned that we can have up to 500 intents, so the size of this response would be ridiculous. You would be quite correct thinking this (though gzip would certainly help here)! Setting the verbose query parameter to false results in a significantly more compact JSON listing.

Once we are ready to deploy into production, we would publish our LUIS app into the Production slot and remove the staging parameter from the URL request. The easiest way to accomplish this would be to simply have your development and test configuration files point at the Staging slot URL and the production configuration to point at the Production slot URL.

You are of course welcome to utilize any other HTTP tool you are comfortable with. In addition, Microsoft provides an easy-to-use console to test the LUIS API within the API documentation found online.²

Extracting Entities

So far, we have developed a simple intent-based LUIS application. But other than it being able to tell our bot a user’s intent, we can’t really do much with it. It is one thing for LUIS to give us information about the fact that the user wants to add a calendar entry, but it better to be able to tell us for what date and time, where, for how long, and with who. We could develop a bot that asks the user for all these details in a linear sequence whenever it sees an AddCalendarEntry. However, this is tedious and neglects the fact that users may very well present the bot with an utterance like this:

It would be a bad user experience to ask the user to reenter all this data. The bot should immediately know what the datetime value of “tomorrow at 6pm” is and that “Huck” is someone who should be added to the invite.

Let’s start with the basics. How do we make sure that “tomorrow at 6pm.” “a week from now,” and “next month” are machine readable? This is where entity recognition comes in. Lucky for us, LUIS comes equipped with many built-in entities that we can add to our application. By doing so, the datetime extraction will “just work.”

If we go back into the Build section of the LUIS App and click the Entities header, we will encounter an empty list of entities (Figure 3-16). We can add three different types of entities. For now, we will simply add a prebuilt entity. We’ll address normal entities and prebuilt domain entities in later sections.

A prebuilt entity is a pretrained definition that can be recognized in utterances. The entity is automatically tagged in the input, and we cannot change how the prebuilt entities are recognized. There is a good amount of logic in them that we can utilize in our applications, and it is best to understand what Microsoft has built before building our own entities.

There are many different prebuilt entities. Not all entities are available across all supported cultures. The LUIS documentation provides details around which prebuilt entities are available across which cultures³ (Figure 3-17).

Some of these entities include what is called value resolution. Value resolution is the process of taking the text input and converting it into a value that can be interpreted by a computer. For example, “one hundred thousand” should resolve to 100000, and “next May 10th” should resolve to 05/10/2019 and so forth.

You may have noticed the JSON result from LUIS included an empty array called entities. This is the placeholder for all entities recognized in the user’s input. A LUIS app can recognize any number of entities in an input. The format of each entity will be as follows:

The resolution objects may include extra attributes, depending on which entity type was detected. Let’s look at the different prebuilt entity types, what they allow us to do, and what the LUIS API result looks like.

Entity Training

Let’s go back into our application and apply some of what we just learned. Being as we are writing an application related to calendars, the most obvious prebuilt entity of choice for us is datetimeV2. On the Entities page, click “Manage prebuilt entities” and select the datetimeV2, as shown in Figure 3-18.

After adding the entity, we should train our model. In the interactive testing UI, when we enter “add calendar entry tomorrow at 5pm,” we should see the result in Figure 3-19.

That was easy. We publish the application to the Staging slot one more time. Using curl to run the same query, we receive the following JSON:

Perfect. We can now utilize datetime entities in any of our intents. This is going to be relevant for us in all our application’s intents, not just the AddCalendarEntry . In addition, we will go ahead and add the e-mail prebuilt entity, retrain, and publish to the Staging slot again. Now we can try an utterance like “meet with szymon.rozga@gmail.com at 5p tomorrow” to get the kind of result we have come to expect.

Custom Entities

Prebuilt entities can do a lot for our models without any extra training. It would be surprising if everything that we need could be provided by the existing prebuilt entities. In our example of a calendar app, calendar entries, by definition, include a few more attributes that we would be interested in.

For starters, we usually want to give meetings a subject (not only “Meet with Bob”) and a location. Both would be arbitrary strings for meetings subjects and locations. How do we accomplish that?

LUIS gives us the ability to train custom entities to detect such concepts and extract their values from the users’ inputs. This is where the power of the entity extraction algorithms really comes in; we show LUIS samples of when words should be identified as entities and when they should be ignored. The NLP algorithms consider context. For instance, given multiple samples of utterances, we can teach LUIS and ensure it doesn’t confuse Starbucks with Starbuck, the character from Moby Dick.

There are four different types of custom entities that we can utilize in LUIS: simple, composite, hierarchical, and list. Let’s examine each one.

Simple Entities

A simple custom entity is an entity such as a calendar entry subject or the prebuilt e-mail, phone number, and URL entities. One segment of the user input can be identified as an entity of said type based on its position in the utterance and the context of the words around it. LUIS makes it easy to create and train these types of entities. Let’s create the calendar subject entity.

Let’s say we want to be clear when we are telling the calendar bot about a subject name for the entry. Let’s say that we want to accept inputs like “meet with Kim about mortgage application at 5pm.” In this example, the subject will be “mortgage application.” Let’s get this in place.

Navigate to the Entities page and click the “Create new entity” button to create a new simple entity called Subject, as illustrated in Figure 3-20.

Once you click Done, the entry is added to the list of entities in your application. The process of training an entity occurs in the same interface as training intents. Let’s navigate into the AddCalendarEntry intent and add the utterance “meet with Kim about mortgage application at 5pm,” as shown in Figure 3-21. Note that this is just a vanilla utterance and no entities are being identified.

We now mouse over the mortgage and application words and notice that LUIS is allowing us to select the words. Click mortgage and then click application so LUIS has the phrase “mortgage application” selected. The pop-up will list all the custom entity types in your application. Select Subject. The utterance in LUIS should now look like Figure 3-22.

Save the utterance and train your app. At this point, LUIS won’t be that great at identifying subjects quite yet. After all, we just provide one example, and entity identification is more difficult to do properly than intent classification. It needs more samples. We can enter a few more utterances in the utterances editor for the add calendar entry intent. A few samples are shown in Figure 3-23.

Note that no subjects at all were identified. Let’s reinforce the concept. It will take quite a few examples for the system to start recognizing the entity. I added more than ten utterances that had some type of subject somewhere in the utterance, as shown in Figure 3-24. Also, be sure to mark the subject of any utterances you may have added yourself. The process of what I call “bending LUIS to your will” can be more of an art than a science. The key point to remember is that there’s going to be an inflection point at which the algorithms start realizing that something following a word is always an entity until some other key words, based on statistical inference. Think of a scale that you are slowly trying to tip into balance. Our utterances should be carefully crafted to ensure we’re capturing as many variations as possible to show LUIS. Often, each variation will also need to include a few samples to really capture the essence of where within the context of an utterance the algorithm can find specific entities.

After training this data set, we see that the interactive testing tool is getting better at identifying the entity. I entered “hi let’s meet about lawn care and harmonicas at 1:45p” (don’t ask how I came up with that…) and received the result in Figure 3-25. We are making good progress. However, if we start entering inputs of different lengths and variations, LUIS may not identify the entities correctly. It just means we need to further train our entity model. We will leave this as an exercise to the reader.

We now have a good grasp of the calendar subject entity even though there are probably many cases that won’t yet work. And truth be told, you won’t be able to capture all the different types of ways users will ask things until you have a good testing phase. That’s how LUIS app development goes. It is worth looking at the resulting JSON when this application is published.

{

  "query": "hi let's meet about lawn care and harmonicas at 1:45pm",

  "topScoringIntent": {

    "intent": "AddCalendarEntry",

    "score": 0.8653278

  },

  "entities": [

    {

      "entity": "1:45pm",

      "type": "builtin.datetimeV2.time",

      "startIndex": 48,

      "endIndex": 53,

      "resolution": {

        "values": [

          {

            "timex": "T13:45",

            "type": "time",

            "value": "13:45:00"

          }

        ]

      }

    },

    {

      "entity": "lawn care and harmonicas",

      "type": "Subject",

      "startIndex": 20,

      "endIndex": 43,

      "score": 0.587688446

    }

  ]

}

Note that the time entity is being identified as expected. The Subject entity comes back with the relevant entity value. It also comes back with a score. The score in this case is again a similar measure to intent scores; it’s a measure of distance from the ideal entity. Unlike intents, LUIS will not return all your entities and their scores. LUIS will return only simple and hierarchical entities with scores above a threshold. For built-in entities, this score is hidden.

The nice thing about training the entity is that even though the samples with the entity are defined in the AddCalendarEntry intent, they carry over to other intents. Intents and entities are not tied directly to each other. I can say “cancel meeting about olympic hockey” and it works as shown in Figure 3-26.

Another observation is the lower score in terms of identifying the DeleteCalendarEntry intent. We’ve added many more utterances to the AddCalendarEntry intent, but DeleteCalendarEntry and EditCalendarEntry have much fewer examples. Take some time to improve that. Add some alternate phrasings and examples with our new Subject entity before we continue.

Exercise 3-4

Training the Subject Entity and Strengthening Our LUIS App

In this exercise, we will improve our LUIS app by training it to do some additional training.

• Add a Subject entity, as per the directions in the previous section.

• Add utterances into your intents to support the Subject entity. Train and test often to see your progress.

• Aim for at least 25 to 30 samples for LUIS to start. Make sure to convey multiple instances of different ways of expressing ideas.

• Ensure all your intents are getting your attention. Make sure every intent has 15 to 20 samples. Include entities in each intent.

• Train and publish the LUIS app into the Staging slot.

• Use curl to examine the resulting JSON.

Training custom entities, especially ones that are a bit vague in terms of positioning and context, can be challenging, but after some practice, you will start seeing patterns in LUIS’s ability to extract them. Note things that need to be explicitly trained: number of words in the subject, subjects with the word and, subjects followed by datetime, and so forth. You may have noticed the explicit mention of number of samples. These are just starting points. An NLU system like LUIS gets better the more sample data it has. Do not overlook this point. If LUIS is not behaving the way you expect it, chances are it is not a LUIS performance problem but rather that your application needs more training.

The second entity we planned to add was the Location entity . Let’s create a new simple custom entity and call it Location. Like the Subject entity, the location is going to be a free text entity, so we’re going to need to train LUIS with many samples.

We’re going to take a stab at this by adding utterances into the AddCalendarEntry intent again. We need to add utterances in these forms:

Meet with kim to talk about {Subject} at {Location}

Meet about {Subject} at {Location}

Add entry with teddy for {Subject} at {Location}

Add meeting at {Location}

Meet at {Location}

Meet in {Location} at {Subject}

You get it. You should also add datetime instances into these utterances. Training the location is going to be trickier as we are teaching LUIS to distinguish between a location and subject, two concepts that simply need a lot of data for LUIS to begin distinguishing since these are two free-text entities. In the end, I ended up adding more than 30 utterances that contained either just a location or a location combined with other entities. After that amount of training, we get decent performance. I can type “meet for dinner at the diner tomorrow at 8pm” and get the following JSON result:

{

    "query": "meet for dinner at the diner tomorrow at 8pm",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.979418

    },

    "entities": [

        {

            "entity": "tomorrow at 8pm",

            "type": "builtin.datetimeV2.datetime",

            "startIndex": 29,

            "endIndex": 43,

            "resolution": {

                "values": [

                    {

                        "timex": "2018-02-19T20",

                        "type": "datetime",

                        "value": "2018-02-19 20:00:00"

                    }

                ]

            }

        },

        {

            "entity": "the diner tomorrow",

            "type": "Location",

            "startIndex": 19,

            "endIndex": 36,

            "score": 0.392795324

        },

        {

            "entity": "dinner",

            "type": "Subject",

            "startIndex": 9,

            "endIndex": 14,

            "score": 0.5891273

        }

    ]

}

We suggest you take some time to strengthen the entities even further. It would be a good experience to really gain an appreciation for the complexities and ambiguities in natural language and in training an NLU system like LUIS.

Exercise 3-5

Training the Location Entity

In this exercise, you will be adding the Location entity into your LUIS app. You will find that this will take a bit longer than the Subject entity by itself.

• Add a Subject entity, as per the directions in the previous section.

• Add utterances into your AddCalendarEntry to support the Location entity. Train and test often to see your progress.

• Aim to start with 35 to 40 samples for LUIS, probably more. As your intents support more entities, you may have to provide more samples to LUIS to properly distinguish. As you add utterances, constantly train and test to see how LUIS is learning. Make sure to use many variations and examples.

• Publish the LUIS app into the Staging slot.

• Use curl to examine the resulting JSON.

This exercise should have been a good experience in strengthening entity resolution when a single utterance contains many entities.

Composite Entities

Congratulations. The work we have done so far is a significant portion of what LUIS can accomplish. Using the intent classification and simple entity extraction techniques described, we can go off and work on our calendar application. Although we went over simple entities, we quickly ran into some complex NLU scenarios. Without a tool like LUIS, doing this kind of language recognition would be incredibly tedious and challenging.

There is another interesting scenario that comes up in natural language. Our model currently supports a user saying a phrase like this:

"Meet at Starbucks for coffee at 2pm"

What if the user wanted to add multiple calendar entries? What if the user wants to say something like the following utterance?

"Meet at trademark for lunch at noon and at Starbucks for coffee at 2pm"

There’s isn’t anything not allowing a user to say that right now. If we’ve trained our app enough, it will certainly handle this input, and it will identify two Subject instances, two Location instances, and two datetime instances, as shown here:

{

    "query": "meet at culture for coffee at 11am and at the office for a code review at noon",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.996190667

    },

    "entities": [

        {

            "entity": "11am",

            "type": "builtin.datetimeV2.time",

            "startIndex": 30,

            "endIndex": 33,

            "resolution": {

                "values": [

                    {

                        "timex": "T11",

                        "type": "time",

                        "value": "11:00:00"

                    }

                ]

            }

        },

        {

            "entity": "noon",

            "type": "builtin.datetimeV2.time",

            "startIndex": 74,

            "endIndex": 77,

            "resolution": {

                "values": [

                    {

                        "timex": "T12",

                        "type": "time",

                        "value": "12:00:00"

                    }

                ]

            }

        },

        {

            "entity": "culture",

            "type": "Location",

            "startIndex": 8,

            "endIndex": 14,

            "score": 0.770069957

        },

        {

            "entity": "the office",

            "type": "Location",

            "startIndex": 42,

            "endIndex": 51,

            "score": 0.9432623

        },

        {

            "entity": "coffee",

            "type": "Subject",

            "startIndex": 20,

            "endIndex": 25,

            "score": 0.9667959

        },

        {

            "entity": "a code review",

            "type": "Subject",

            "startIndex": 57,

            "endIndex": 69,

            "score": 0.9293087

        }

    ]

}

And yet, parsing this using code would be quite challenging. How do we tell which entities should be grouped together? Which location goes with which subject? You should be able to use the startIndex property to figure it out I suppose, but that’s not always as obvious.

Lucky for us, LUIS can group the entities into what are called composite entities . Rather than the messy result shown previously, LUIS will tell us which entities are part of which composite entity. This makes it way easier for us to know that there were two separate AddCalendar requests, one for 11 a.m. coffee at Culture and another one for a code review in the office at noon.

Composite entities can be created on the Entities page of LUIS. Figure 3-27 illustrates the process. Click the Create new entity button, enter a name for the entity, select the Composite entity type, and select the child entity types to be included as part of the new entity. We will use the name CalendarEntry to identify our composite entity.

Once it is created, we need to properly train LUIS to recognize it. Let’s look at the AddCalendarEntry intent again. The easiest way to train LUIS would be to find all utterances that have the required three entities and wrap the entities into the composite entity. Figure 3-28 shows an example.

Click the first Location entity . A pop-up will appear asking you to relabel the entity or wrap it in a composite entity. Click “Wrap in composite entity” (Figure 3-29).

We move our mouse over the Subject and datetimeV2 entities . Note the green underline expands to cover each entity (Figure 3-30). Click datetimeV2 so that it is included in the composite entity and click the CalendarEntry name.

Do the same for the second instance of the CalendarEntry entity . The result should look like Figure 3-31.

We should do the same for any other utterance we can find that includes the three entities. Once we train and publish the app, LUIS should start extracting this composite entity. We only show the relevant API section here:

"compositeEntities": [

    {

        "parentType": "CalendarEntry",

        "value": "culture for coffee at 11am",

        "children": [

            {

                "type": "builtin.datetimeV2.time",

                "value": "11am"

            },

            {

                "type": "Subject",

                "value": "coffee"

            },

            {

                "type": "Location",

                "value": "culture"

            }

        ]

    },

    {

        "parentType": "CalendarEntry",

        "value": "the office for a code review at noon",

        "children": [

            {

                "type": "builtin.datetimeV2.time",

                "value": "noon"

            },

            {

                "type": "Subject",

                "value": "a code review"

            },

            {

                "type": "Location",

                "value": "the office"

            }

        ]

    }

]

Exercise 3-6

Composite Entities

In this exercise, you will add composite entities to your LUIS app.

• Create a composite entity called CalendarEntry, composed of datetimeV2, Subject, and Location entities.

• Train every utterance that has these three entities to recognize the composite entity.

• Train additional examples with multiple instances of the CalendarEntry composite entity . Remember, it takes time, dedication, and persistence to get it right.

• Publish the LUIS app into the Staging slot.

• Use curl to examine the resulting JSON.

Composite entities are a great feature to group entities into logical data objects. Composite entities allow us to encapsulate more complex expressions.

Hierarchical Entities

A hierarchical entity allows us to define a category of entities and its children. You can think of hierarchical entities as defining a parent/subtype relationship between entities. We have run into this type before. Do you recall the Datetimev2 entity? It had seven subtypes such as daterange, set, and time.

LUIS allows us to easily create our own subtypes. Say we wanted to add support in our model to specify the calendar entry visibility as public or private. We could add support for utterances like this:

"create private entry for interview with competitor at starbucks"

 "create invisible entry for interview with recruiter at trademark"

The words private or invisible here indicate the visibility field of the calendar. Why would we create a hierarchical entity as opposed to a simple entity? Can’t we just look at the value of a Visibility property and determine whether it should be a private meeting or not? Yes and no. If the user sticks to those two words, yes. But remember, natural language is ambiguous and vague. Phrasings change. The user can say invisible, private, privately, hidden. It’s the same with public. If we make assumptions about a closed set of options in our code, then we would have to change our code any time a new option shows up. The reason a hierarchical entity should be used as opposed to a simple one is that the statistical models of where in context the hierarchical entity appears is shared by the subtypes. Once that is identified, the step of identifying the child entity is essentially a classification problem. Making the entity hierarchical makes for better LUIS performance versus two simple entities. Not to mention, it’s more efficient to have LUIS classify the meaning of an entity in the context of our application rather than writing code to do so.

Figure 3-32 illustrates the creation of a new hierarchical entity. We do this by visiting the Entities page, clicking “Create new entity,” and selecting Hierarchical from the entity type drop-down. We give the parent entity a name and add the child entities. Once we click Done, it is a matter of going into the intent utterances and training LUIS. Let's go into AddCalendarEntry and add a few samples.

You may notice that one or two samples are not sufficient. We need to give LUIS a really good idea of where and how it may encounter the public and private visibility modifiers before it can start recognizing the entity in our inputs. The ten samples in Figure 3-33 were a good start.

Once we train and publish , we can view the resulting JSON via curl, as shown here:

{

    "query": "create private meeting for tomorrow 6pm with teddy",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.9856489

    },

    "entities": [

        {

            "entity": "tomorrow 6pm",

            "type": "builtin.datetimeV2.datetime",

            "startIndex": 27,

            "endIndex": 38,

            "resolution": {

                "values": [

                    {

                        "timex": "2018-02-19T18",

                        "type": "datetime",

                        "value": "2018-02-19 18:00:00"

                    }

                ]

            }

        },

        {

            "entity": "private",

            "type": "Visibility::Private",

            "startIndex": 7,

            "endIndex": 13,

            "score": 0.9018322

        }

    ]

}

{

    "query": "create public meeting with jeff",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.975892961

    },

    "entities": [

        {

            "entity": "public",

            "type": "Visibility::Public",

            "startIndex": 7,

            "endIndex": 12,

            "score": 0.6018059

        }

    ]

}

List Entities

So far, the prebuilt, simple, composite, and hierarchical entities were all extracted from user input via machine learning techniques . Every time we added one of these entities and trained LUIS , you may have noticed the number of models being trained increased. Recall that a LUIS application is composed of one model per intent/entity. By now, we should be at ten models. Each of these is rebuilt any time we train our app.

List entities exist outside this machine learning world. A list entity is simply a collection of terms and synonyms for those terms. For example, if we want to identify cities, we can add an entry for New York that has the synonyms NY, The Big Apple, The City That Never Sleeps, Gotham, New Amsterdam, etc. LUIS will resolve any of these alternate names into New York.

Once a custom list entity type is created, we are redirected to a list entity editor in which we can enter the canonical term and the synonyms. This interface allows us to add new terms and their synonyms. It also makes recommendations to add extra terms that seem related to what we have added thus far. List entities are limited to 20,000 terms, including synonyms. We can have up to 50 list entities per application, so there is a lot of potential for LUIS-based term and synonym lookup features. Figure 3-34 shows a sample custom list entity definition.

Since list entities are not learned by LUIS, new values are not recognized based on context. If LUIS sees “Gotham,” it identifies it as New York. If it sees “Gohtam,” it does not. It is literally a lookup list.

{

    "query": "meet in the big apple",

    "topScoringIntent": {

        "intent": "AddCalendarEntry",

        "score": 0.943692744

    },

    "entities": [

        {

            "entity": "the big apple",

            "type": "Cities",

            "startIndex": 8,

            "endIndex": 20,

            "resolution": {

                "values": [

                    "New York"

                ]

            }

        }

    ]

}

When using the API, LUIS will highlight the term that matches a list entity type and will return the canonical name in the resolution values. This allows your consuming application to ignore all the possible synonyms for a term and execute logic based on the canonical names. List entities are powerful for situations where you know the set of possible values for terms ahead of time.

Prebuilt Domains

I hope you have gained an appreciation for some of the challenges of building NLU models . Machine learning tools allow us to get computers to start learning, but we need to be sure we are training them with a lot of good data. It takes years of day-to-day interactions for humans to be immersed in a language to be able to truly understand it. Yet, we assume that AI means that a computer will be able to pick up the concepts with ten samples. When it doesn’t, sometimes we think to ourselves, “Oh, come on, you should know this by now!”

To help us on our journey, many of the NLU platforms provide what are called prebuilt models or domains . Essentially, the creators of LUIS and other platforms want to give us a head start with some domains that we can easily include in our application, train LUIS, and be off to the races. Some of LUIS’s prebuilt models are shown in Figure 3-35.

We can find prebuilt domains in LUIS by navigating into the Build section and clicking the Prebuilt Domains link in the bottom left. At the time of this writing, this feature is still in Preview mode. That is the reason it is so isolated and why it is dynamic and may change by the time you read this. LUIS includes a variety of domains from Camera to Home Automation to Gaming to Music and even Calendar, which is similar to the app we have been working on in this chapter. In fact, we will do just that in Exercise 3-7. The “Learn more” text links to a page that describes in detail what intents and entities each domain pulls in and which domains are supported by which cultures.⁴

When we add a domain to your application, LUIS will add all the domain’s intents and entities into our application, and they will count toward the application’s maximums. At that point, we able to modify the intents and entities as we see fit. Sometimes you may want to get rid of certain intents or add new ones to complement the prebuilt ones. Other times we may need to train the system with more samples. We suggest the prebuilt domains are treated as starting points. Our goal is to extend them and build great experiences on top of them.

Phrase Lists

So far, we have been exploring different techniques to create great models. We have the tools we need to make sure we can create a good conversational experience for our users. There are cases when we train LUIS that the model performance is not as good as we would like. Entities may not be getting recognized as well as we would like them to. Maybe we are building a LUIS app that deals specifically with internal terms that aren’t exactly part of the culture your application is using. Maybe we haven’t had a chance to train LUIS entities with every known possible value for an entity and list entities don’t cut it because we want our entities to remain flexible.

One way to improve LUIS performance under these circumstances is to use phrase lists. Phrase lists are hints, rather than strict rules, that LUIS uses when training our app. They are not a silver bullet but can be very effective. A phrase list allows us to present to LUIS a category of words or phrases that are related to each other. This grouping is a hint to LUIS to treat the words in the category in a similar way. In the case of an entity value not being recognized properly, we could enter all the known possible values as a phrase list and mark the list as exchangeable, which indicates to LUIS that in the context of an entity, these values can be treated in the same way. If we are trying to improve LUIS’s vocabulary with words it may not be familiar with, the phrase list would not be marked as nonexchangeable.

Let’s say we wanted to improve our Calendar model’s private visibility entity performance. After all, there are many ways of expressing that we want a private meeting. As a starting point, we could add a phrase list with all the different words we could expect the model to see. Figure 3-36 shows the LUIS user interface for working on a phrase list. You can get here by selecting the Phrase Lists item under the Build page and clicking Create new phrase list.

A phrase list requires a name and some values. We enter the values one by one in the Value field. As we press Enter, it adds them to the Phrase list values field. The Related Values field contains synonyms automatically loaded by LUIS. We then select the checkbox to tell LUIS that the values are interchangeable.

Before training, let’s try a few variations of the private meeting utterances without the phrase list enabled. If you try utterances like “Meet in private,” “Meet in secret,” or “Create a hidden meeting,” LUIS does not recognize the entity. However, if we train the app with the phrase list, LUIS has no problem identifying the entity in those samples and many others.⁵

If you have completed this exercise successfully, congratulations! You are getting darn good at using LUIS.

Active Learning

We’ve spent weeks training a model , we’ve gone through a round of testing, we’ve deployed the application into production, and we’ve switched our bot on. Now what? How do we know if the model is performing the best it can? How do we know whether some user has thrown unexpected input at the our application that breaks our bot and results in a bad user experience? Bug reports are one way for sure, but we would depend on getting that feedback. What if we could find out about these problems as soon as they occur? We can do so by taking advantage of LUIS’s active learning abilities.

Recall that supervised learning is machine learning from labeled data, and unsupervised learning is machine learning from unlabeled data. Semisupervised learning lives somewhere in between. Active learning is a type of semisupervised learning in which the learner asks the supervisor to label new data samples. Based on the inputs that LUIS is seeing, it can ask you, the LUIS app trainer, for your assistance labeling data that is coming from your users. This improves model performance and over time makes our application more intelligent by using real user input as sample data.

You can access this functionality through the Review endpoint utterances link on the Build page (Figure 3-37). Throughout the training of the application, we’ve been utilizing the published application endpoint to test various utterances. LUIS bases its active leaning on the inputs against the endpoint, not the Interactive Test feature.

The interface allows us to review past utterances and their top-scoring intent, referred to as the aligned intent . As trainers, we can add the utterance to the alignment intent, reassign to a different intent, or altogether get rid of the utterance. We can also zero in on specific intents or entities if we know there are problems with any of them.

Before adding the utterance to the aligned intent, we need to confirm that the utterance is correctly labeled and any entities are being correctly identified. We suggest that using this interface to improve LUIS application is a common practice for any team .

Dashboard Overview

Now that we have trained our application and utilized it for testing, it is well worth highlighting the data that the dashboard provides. The dashboard allows us to get a good glance at the overall app status, its usage, and the amount of data we have trained it with.

The very top provides information about the last time we trained and published the application, as per Figure 3-38. We can also get some metrics about the number of intents and entities we are using, the number of list entities we have, and how many total labeled utterances our application has so far.

The next section displays the kind of usage that the application is getting through the API. We can monitor the amount of endpoint hits for the last week up to the last year. This data is available only once an application is published to the production slot. This is illustrated in Figure 3-39.

Lastly, we are presented with an intent and entity breakdown, as shown in Figure 3-40. Here we see a distribution of the percentage of utterances used to train each intent. You can clearly see some of our intents contain significantly more sample utterances than others. It’s the same for entities. The uneven distribution does not necessarily mean that an entity or intent needs more training.

Managing and Versioning Your Application

Everything we have done so far is part of the common workflow of adding samples, training, and publishing a LUIS application. During the development phase, this workflow is repeated over and over again. Once your application is in production, you should be careful about what you do to your app. The process of adding a new intent or entity can have unforeseen effects on the rest of the application, and it is best that editing an existing application is done in isolation so it can be tested properly.

We have experience with the concepts of the staging and production deployment slots. This certainly helps; we know that we can test changes without publishing to our production endpoints. A common rule is to have the Staging slot host the dev/test version of the application and the Production slot host the production version. Whenever a new application is ready for production, we move it from the Staging slot to the Production slot. But what if we make a mistake in our models? What if we need to roll the Production slot back? That is where versions come in.

LUIS allows you to create a named version of the application at any point in time. So far, by default we have been working on version 0.1. Once it is ready for production, we can publish it and clone it into a new version 0.2. At that point, you set the 0.2 version to Active. Now, the LUIS interface is editing version 0.2. If we accidentally publish version 0.2 into the production slot, we can easily go back to version 0.1 and publish that. Once version 0.2 is production ready, we deploy that into the Production slot and clone it into version 0.3 and set that version as the active version. And so forth. If at any point you deploy a version into the Production slot and need to revert, you set your LUIS active back to 0.2 and publish that version into the Production slot. The workflow is illustrated in Figure 3-41.

We access the application version information through the Settings page. Figure 3-42 and Figure 3-43 show the interface plus what it looks like after cloning version 0.1 into 0.2.

Note that after closing 0.1, it remains in the Staging slot, but 0.2 becomes the Active version. LUIS also doesn’t allow for easy branching. If multiple users want to make changes to a single version, they cannot create a new version and then merge their changes using the user interface. One way to accomplish this would be to download the LUIS App JSON by clicking the Export Version button in Figure 3-42, utilizing a source control tool like Git to branch and merge, and finally, using the “Import new version” button to upload a new version from a JSON file.

The same page also allows us to add collaborators to the application. This is a great way to give access to other folks in your organization to assist in editing, training, and testing versions of the app. At the time of this writing, there are no fine-tuned audit controls; all collaborators can do anything with the application except add/remove other collaborators (Figure 3-44).

Integrating with Spell Checking

One advanced feature in LUIS is the ability to integrate with a spell checker to automatically fix misspellings in user input. User input is, by its very nature, messy. Misspellings are immensely common. Combine that with the common usage of messaging apps, and you have a recipe for consistent misspelled input.

The spell checker integration runs the user query through Bing’s Spell Checker service, gets a possibly altered query with misspellings fixed, and runs that altered query through LUIS. This feature is invoked by including the query parameters spellCheck and bing-spell-check-subscription-key. You can get a subscription key from the Azure Portal, which we will introduce in Chapter 5. We will also utilize the Spell Check API more directly in Chapter 10.

This functionality can be helpful, and we would typically recommend it with a caveat. If our entities contain domain-specific values or product names that are not strictly part of the English language, we may get an altered query in which LUIS is unable to detect an entity. For example, it may break up one word into multiple words when such behavior is unwanted. Or, if our application is expecting financial tickers, it may just change them. For example, VEA, a Vanguard ETF , is changed to VA. In the United States, that’s a common reference to the state of Virginia. The loss of meaning is quite significant; I advise caution in using this feature.

The effect of the spell check on the LUIS API result is easy to spot. The result now includes a field called alteredQuery. This is the text passed into the LUIS models. A sample curl request and response JSON is presented here:

Import/Export Application

Any application built in LUIS can be exported into a JSON file and imported back into LUIS . The JSON file format is exactly what we would expect. There are elements that define which custom intents, custom entities, and prebuilt entities the application uses. There are additional elements to capture phrase lists. And, not surprisingly, there is a rather large segment describing all the sample utterances, their intent label, and the start and end index of any entities in the utterance. We can export the application by clicking Export App in the My Apps section of LUIS or Export Version in the Settings page, as per Figure 3-41.

Although the format of the exported application is specific to LUIS, it is easy to imagine how we could write code to interpret the data by other applications. From a governance perspective, it is good practice to export our applications and store the JSON in source control because the action of publishing an action is irreversible. This should not be an issue if our teams follow a strategy in which a publish into the Production slot implies the creation of a new application version, but mistakes do happen.

One of the most common questions we receive in our work with LUIS is “why we can’t import an application into an existing application?” The reason is that this would be tantamount to a smart merge, especially where there are overlapping utterances with different intents or same name intents with completely different application connotations. Since every application has different semantics, this merge would be a nontrivial task. We suggest either utilizing Git to manage and merge application JSON code or creating custom code to merge using the LUIS Authoring API.

Using the LUIS Authoring API

The API is very rich and allows for training, custom active learning, and enables CI/CD type scenarios. The API Reference Docs⁶ are a great place to learn about the API.

Troubleshooting Your Models

Building LUIS applications can be more of an art than science. You will sometimes spend a lot of time teaching LUIS the difference between some entities or where in a sentence an entity can appear. Be patient. Be thorough. And always think of the problem in statistical terms; the system needs to see enough samples to truly start understanding what’s happening. As people, we can take our intelligence and language understanding for granted. In relative terms, it is quite amazing how quickly we can train a system like LUIS. Remember this as you work with LUIS or any other NLU system.

Conclusion

That was quite a lot of information! Congratulations, we are now equipped to start building our own NLU models using a tool like LUIS. To recap, we went through the exercise of creating an application by utilizing prebuilt entities, custom intents, and custom entities. We explored the power of the various prebuilt entities and dabbled a bit in the prebuilt domains that LUIS provides. We spent time training and testing our application, before publishing it into different types of slots and testing the API endpoints using curl. We optimized our application using phrase features and further improved it by using LUIS’s active learning abilities. We explored versioning, collaborating, integrated spell check, exporting and importing of applications, using the authoring API, and common troubleshooting techniques in our LUIS applications.

I must reiterate that the concepts and techniques you just learned are all applicable to other NLU platforms. The process of training intents and entities and optimizing models is a powerful skill to have in your toolkit, whether for bots, voice assistants, or any other natural language interface. We are now ready to start thinking about how we build a bot. As we do, we’ll keep checking back into this LUIS application as it gets consumed by our bot.