IBM Watson Assistant

In Watson Assistant, dialog skills combine intents and entities in a dialog tree to understand and respond to end user queries. Through the manual creation process, when an Assistant user creates an intent, they must manually author alternative phrases their end user might input. This is time consuming and lacks accuracy because it often does not capture the wide range of ways end user may ask the same questions.

To address this issue, the Assistant team developed an intent recommendations feature which leverages chat logs to automate the intent creations process. Assistant users can upload chat logs and similar utterances within these logs will be clustered to form intent recommendations. These recommendations can be turned into need intents, or added to existing intents.

To address the issue of accommodating both new and existing users, empathy and journey mapping were done for both user types in an effort to ensure that both user groups would have their unique needs met through an intuitive flow. Existing users still needed to be able to easily find intents they had already created, while being introduced to the new intent recommendations option and understanding how they can use this feature to make their training data more robust. Meanwhile, new users needed to be informed of both creation methods when they land on the intents page so that they can select the option they wish to start with, and be aware of how they can leverage both methods to generate training data for their system.

To address the challenge of users discovering the intent recommendations feature, a standard Watson Assistant banner was used. This is a common component used in the UI to highlight new features. In this instance, the empty state of the banner was used provide a brief description of the new recommendations feature and provide a link to learn more or upload files. Once chat logs have been ingested, the banner serves as a preview, displaying the top four intent recommendations.

To confirm user comprehension of the new feature, we set up a user test with IBM sponsor users. Some had worked with intents before, and some had not. In this test, participants were given a scenario in which they were to act as though they were an employee of a large telcom tasked with setting up an FAQ bot for the company. We provided them with a .csv file of chat logs from this imaginary telcom and asked them to build some entities. The main research questions from the test were ifs the feature discoverable, if users understood how to use the feature, and if they found value in this new method on intent creation.

Based on the user test, we were able to determine that participants were excited about the new feature and thought it could be a useful tool both during initial set-up, and to improve a more mature bot. Participants were able to understand the concept and upload the provided chat log easily. However, some findings uncovered room for improvement, which were addressed with immediate fixes and plans for longer term improvement. The documentation was revised to provide more information about how the chat log data should be structured. Top intent recommendations are now highlighted based on uniqueness and frequency of the utterances in the recommendation. This gives users more clear guidance on where they can start adding intents to have the greatest impact on their training data. Additionally, the ability to rename intent recommendations before adding them to training data was implemented and we worked on future plans to integrate more dynamic ways to explore recommendations such as searching and combining recommended intents.