While auto-testing in Teneo allows you to perform quality assurance tasks during the development and maintenance of a Teneo solution, you may wish to perform end-to-end tests before you release your bot to ensure the bot behaves as intended, the answers provided are as expected and the integrations work as anticipated.
Botium is a quality assurance framework that was specifically developed for regression testing of chatbots, including Teneo chatbots. Botium is a third party product developed by Botium.ai.
A webinar on Botium can be found here
A Botium test project typically contains the following components:
This page will guide you through setting up a Botium project and run tests against a Teneo chatbot.
You will need access to a Botium server, otherwise known as a "Botium Box". If you don't have access to Botium, you can sign up for a free trial on the Botium.ai website. Don't forget to mention that you are coming from our page!
The Teneo Connector is included in the Professional version of Botium. It is not included in the free version.
You need to know the URL of the published Teneo solution that you want to test. If you haven't published your solution yet or don't know the URL of your bot, see Publish your bot to learn more.
Make sure you've logged in to your Botium instance through your favorite browser before you follow the setup instructions.
First, we are going to connect a Teneo chatbot to the Botium instance.
In the settings tab of your chatbot configuration, you can specify static input parameters that will be included in every Botium request your bot. For example, here you could specify a parameter that tells Teneo that the requests are coming from Botium. You can specify multiple static input parameters per chatbot.
Now we are going to set up a test set in which the test cases are going to be stored.
To create a test case, we're going to use Botium's live chat functionality which will allow you to have a dialog with your chatbot directly from within Botium and save that dialog as a test case.
The record live chat functionality is available from multiple views in Botium, you can find them here:
Let's set up a test project that ties your chatbot, test set, and test case together. We're going to use the 'Quickstart' wizard which allows us to select which chatbot we want to test against, and which test set(s) that we want to bind to the test project.
In the second step, you'll assign the test set(s) that you want to bind to the test project.
In the last step, we'll leave everything at default, and click 'Save'.
You've now set up a test project that contains:
Note that the test project is a static entity, you cannot add additional test sets after a test project has been set up.
You can start testing your chatbot directly from the test project by clicking on the 'Start Test Session Now' button. Once you’ve started the test session, Botium will present the test results. Here you can see which test cases failed and which passed. You can also expand the test cases to inspect the dialogs.
If you'd like to run additional tests, navigate back to your test project and click 'Start test session now'.
The free version only allows 50 tests per day
Botium provides plenty of information when a test case fails:
Note that Botium aborts the test case as soon as the test case fails. It will continue testing the next test case if available.
It's possible to get even more information per transaction by clicking on the <> symbol found next to the user inputs and the chatbot's responses. This will open Botium code which allows you to see in detail what was sent to and received from your chatbot.
In addition to testing, if the bot gave the correct answer for a particular input, it's important that we can test all parts of a chatbot to make sure they are functioning as expected. This includes extra parameters that may be sent to the bot or additional details that are included in the bot's response.
We will be using Botium's 'Source editor' to explain some of the advanced testing options in Botium. You can modify the source code of a test case by opening a test case and clicking the 'Open in source editor' at the bottom of the test scenario.
The source code of a test case typically looks like this:
User wants to know the time
#me
Hello!#bot
Hello. It's good to see you!#me
What time is it?#bot
My watch says it's time to get some exercise!
To correctly test the behavior of our chatbots, we may have to send input parameters. We add the input parameters below the user inputs (#me) like this:
#me
Hello!
UPDATE_CUSTOM QUERY_PARAM | callerCountry | GB#bot
Hello. It's good to see you!
Let's inspect the part that is sending the input parameter:
The UPDATE_CUSTOM QUERY_PARAM
functions tell Botium to send input parameters followed by a key and value pair, separated by pipes. If you want to send multiple input parameters from the same user input, you will have to create a new line for each input parameter, using the same syntax.
Below is an image that illustrates where input parameters, that were included in requests, appear in the engine response JSON:
Sometimes you might want to evaluate parameters that are included in the engine response JSON, like output parameters:
To test parameters included in the JSON response from Teneo, Botium offers a JSON path asserter. An example scenario that asserts the value of an output parameter called OUTPUT_NODE_IDS may look as follows:
#me
Hello!#bot
Hello. It's good to see you!
JSON_PATH $.output.parameters.OUTPUT_NODE_IDS | 0581322c-e766-4b47-8de2+c4a1169a787d
Let's inspect the part that checks the JSON path:
To evaluate the returned JSON, you use the JSON_PATH
function along with:
Note that in the example test scenario above, Botium will assert both the bot's text response (Hello. It's good to see you!) and an output parameter called OUTPUT_NODE_IDS. If you'd prefer to just test a JSON path, you can omit the expected text response. If you want to test multiple JSON paths in the same engine response, you should create a new line for each path, using the same syntax.
It's important to make sure that the links that your chatbot is returning are working as intended. To do this, you can use the link checker that's available in Botium.
#me
What drinks do you serve?#bot
We serve everything from flat whites to espressos. Please visit https://longberrybaristas.ai/menu/ to see the full menu.
CHECKLINK 200
The CHECKLINK
function validates URL's available in both the output text as well as the output's URL field. When testing the URL, Botium asserts if the HTTP response code it received, matched the value specified in the test scenario.
Botium offers many more useful features for testing your bot and we encourage you to browse and explore Botium's own documentation to make the most of Botium.
Here is a list of some of the useful features that might be beneficial when you are composing tests for your Teneo chatbot:
Was this page helpful?