Documentation

• 1: Getting Started
• 2: Converting NLU Models to ServisBOT Common Format
• 3: EULA

1 - Getting Started

Prerequisites

Docker is required to run this unsupervised learning trial.

Docker Settings

After Docker is installed and running, navigate to “Preferences” by clicking the gear icon on the top right of the screen. Then navigate to resources. We recommend adjusting the “Memory” to 8.00 GB. Once adjusted, click “Apply & Restart”.

Installation without CLI

To use CLI commands for installation and use, please scroll down to “Installation with CLI”.

We recommend the CLI method for installation and use.

To run the insights tool without the use of the CLI, open the Docker Desktop application installed on the previous step.

Navigate to the “Add Extensions” tab on the left side of the screen.

Search for an extension named “Portainer” and install it. This should only take a few seconds.

When it is installed the extension will appear on the left side of the screen.

Open Portainer and navigate to the “Home” tab. Then click “local” under “Environments.”

Navigate to the “Containers” tab and click on “Add container” at the top right of the screen.

This step is where the container is created in which the insights tool will run.

Name your container “insights” at the top. For “Image” paste in:

servisbot/insights

If the image name is not spelled correctly no image or an incorrect image will be pulled.

Navigate to “+ publish a new network port” below and click it twice.

Two rows will appear with headers “host” and “container.” The rows should look like this:

host:80     container:80
host:3000   container:3000

From there click on “Deploy the Container.” Make sure nothing else has been altered on the page, as it could affect the installation of the insights tool.

A reference image is below. This installation from Docker Hub may take a few minutes to complete. The most recent version of the insights tool will be installed and run on your Docker Desktop app when this process is complete.

The deployed container should look like the image below.

When the container is deployed and running, navigate to localhost in your browser.

If these steps have been completed, the next two steps, “Installation with CLI” and “Setup with CLI” can be skipped. Please navigate to “Running the Tool.”

Installation with CLI

The latest version can be found on Docker Hub.

To pull the current release, after installing docker, use your terminal to run the following command:

docker pull servisbot/insights

Setup with CLI

Once the container is downloaded you can run it with the command:

docker run --rm -p 80:80 -p 3000:3000 -i servisbot/insights

Once it is running, navigate to localhost in your browser to start with the experience.

Running the Tool

Login

To login, use the same business email address used when registering for the trial. The trial will allow running the service using ServisBOT sample data until you have a working API key.

If you choose to upload your own data, you will need to request an API key from this page, using the same business email address you provided when registering for the trial.

Select Your Demo

Once signed in, the automated machine learning Insights tool is ready to be used. Select “General Step 1 NLU Automation Demo” to get started.

From here, an outline of the NLU automation process is shown.

Hitting “Next” will navigating you to the next screen and prompt you to upload a bot design.

Upload Your Designs

It is important to make sure that files are formatted correctly.

In order to run the service, the data should be in the ServisBOT common format, we do however support automatic conversion to this format with a number of supported export formats which can be uploaded directly. An error message will appear if the file is not formatted correctly.

ServisBOT Common Format

In the ServisBOT common format, data is arranged by the required columns “Botname”, “intent”, and “utterance”. The order of the columns does not matter but names are case sensitive. For botnames and intents with multiple words, avoid using spaces as separators. Recommended special characters for word separators include hypens “-” and underscores “_”. PascalCase or camelCase are also recommended for compound bot or intent names. In the “utterance” column any character can be included, including emojis.

To convert your data into the ServisBOT common format, see converting to ServisBOT Common Format.

Analyze Data

When the analysis is complete and displays 4 checkmarks this signifies that a report has been generated for the bot design outlining some general statistics of the dataset. It is important to note that at this stage no optimization has been performed.

Once the analysis is complete and there are 4 checkmarks, a report has been generated for the bot design outlining some general statistics of the dataset. It is important to note that at this stage no optimization has been performed.

Navigating to the next page will show a scorecard of the results.

Review Results

This page shows initial accuracy scores of the bot design without any ServisBOT optimization. The scorecard is shown below the loading bar.

Each item on the scorecard will reveal a definition of the term if hovering over it with the mouse.

• Baseline Accuracy: The probability of identifying an intent correctly given a confidence threshold
  • The confidence threshold is set at 0.6 (60%) for this tool
• True Positives: The number of utterances correctly classified above the confidence threshold
  • An utterance must match to an intent with a confidence of 60% or higher
• False Positives: The number of utterances incorrectly classified
• False Negatives: The number of utterances not classified due to low confidence

• Total Utterances: The total number of utterances in the dataset
• Training Utterances: The number of utterances used to train the model
  • Roughly 80% of the data
• Testing Utterances: The number of utterances used to test the model
  • Roughly 20% of the data
• Topic Count: The number of topics identified after clustering the data

Intents are split into categories of “Good” and “Bad”.

• Good intents are defined by having more than 5 utterances with those utterances being closely related to one another.
• Bad intents are defined by being undertrained, poorly defined, or having bad samples.
  • Undertrained: An intent with fewer than the minimum number of samples required to properly test an intent (fewer than 10 utterances)
  • Poorly Defined: An intent with fewer than the minimum number of samples to properly define an intent (fewer than 3 utterances)
  • Containing Bad Samples: An intent containing utterances in command form or longform
    • Command: an utterance containing 3 or fewer words
    • Longform: an utterance containing 20 or more words

While reviewing the scorecard displayed above, a subsequent step in the NLU Automation process, dataset optimization, is running behind the scenes. The goal of dataset optimization is to produce an improved model.

Once the process is complete, navigate to the next screen by clicking the “View Updated Model” button.

Improvement results will be shown along with a side by side comparison of the existing model and the updated model.

Above the scorecards there are “View Accuracy Details” and “View NLU Performance” buttons.

Clicking either button will navigate to a new page for their respective details.

The “View Accuracy Details” button brings up a classification report that shows intents broken down by number of true and false positives, precision, recall, and f1 scores. The table can be sorted by column by clicking the column header. The “Display Comparison” slider on the top right will overlay the existing and new model for a side-by-side comparison.

The “View Model Comparison” button below the expanded classification report will navigate back to the side by side comparison of the models.

The “View NLU Performance” button navigates to a screen with four graphs showing comparisons of the existing and new models. More details about each distribution can be found by hovering over different points on the graphs.

The “View Model Comparison” button below the graphs will navigate back to the side by side comparison of the models.

Once all results have been reviewed, the end of the process has been reached.

The entire process can be started over with new data by clicking “Start Over” button at the bottom of the screen on any of the results pages.

For more information about the tool, customization, or to leave feedback, click the link in the top right corner of any page within the Insights tool.

Personally Identifiable Information (PII)

For more information about ServisBOT, click the link in the top left corner of any page within the Insights tool to navigate to our website.

2 - Converting NLU Models to ServisBOT Common Format

To get started with an NLU model stored in a public NLU vendor, follow the instructions below.

Cognigy

From the Cognigy console, select the bot from the list of agents.

From there, navigate to the menu on the left side of the screen and click “Manage”. A dropdown menu will open up. Select “Packaging”.

Select the lexicons(intents) to be exported.

From there click “Next”. The next screen contains a text box for a “Package Name”. It will be labeled “Package-BOTNAME”. This can be kept as is, or changed to remove “Package” from the beginning. Click “Export” then click the link next to “Download Export”.

This will result in a .zip file being downloaded with a name similar to “BOTNAME_DATE_TIME.zip”. When the .zip file is opened a root folder will contain a folder labeled “lexicon”. The files in the folder are labeled with an id. Each id represents an intent name, which can be found by opening each file.

Make sure to note the id associated with each intent. The intent name can be found next to “name”.

Utterance files can be found in the folder labeled “lexiconKeyphrase”. The files are also labeled by id. They are not grouped specifically by intent.

After opening the file, the utterance can be found next to “value”. To find out which intent the utterance belongs to, find the id next to “lexiconReference”.

Create a CSV file with the following columns: Botname, intent, and utterance.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

Since each utterance is located in its own file, this process may take a long time, expecially for larger bots. In this case, it may be easier to create a csv by referencing the Cognigy console.

Navigating to “Lexicons” under “Build” on the menu on the left of the screen will bring up the intents for the bot. Clicking on each intent will bring up the list of utterances for that intent.

DialogFlow CX

Bot data is normally exported and a .json file is downloaded containing the bot name, intents, and utterances. However, for DialogFlow CX that option is not supported and the conversion of the data into ServisBOT common format has to be done directly from the DialogFlow CX console.

From the console, select the project and then select a bot from the “Agents” list. Each agent represents a bot.

From there, navigate to the “Manage” tab on the menu on the left side of the screen. Navigate to “Intents”. Each intent will be displayed.

Clicking in to each intent will show the utterances of that intent.

Use the intent data to create a CSV file with the following columns: Botname, intent, and utterance.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

DialogFlow ES

If your exported bot fails to parse, you can follow these manual steps to convert your DialogFlow ES bot to the ServisBOT common format.

From the DialogFlow ES console, open the menu by clicking the three lines icon on the top left of the screen. Select a bot from the drop-down menu and then navigate to “Settings” by clicking the gear icon next to the drop-down and bot name.

Click on the “Export and Import” tab and select “Export as Zip”.

Exporting a DialogFlow bot design will download a zip file. After unzipping the file there will be a folder with the bot name. In the folder there will be 2 .json files and an “intents” folder. Within the intents folder are 2 kinds of files - files with utterances and files with intent names, all .json. Utterance files contain an intent name followed by usersays and the language. Intent files just say the intent name.

The INTENT.json file will look something like the above image, where “name” corresponds to the intent name.

The INTENT_usersays_.json will look the the above image, where the list “data” will contain “text” which corresponds to an utterance.

Create a CSV file with the following columns: Botname, intent, and utterance.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

Lex V1

If your exported bot fails to parse, you can follow these manual steps to convert your Lex V1 bot to the ServisBOT common format.

When exporting a bot design from Lex V1, there are 2 platform options, Alexa Skills Kit & Lex. Each produce a different format of data.

In the LexV1 console, select your bot from the list and click “Actions.”

A dropdown menu will appear. Select “Export”. After selecting a bot version, a platform can be selected, Amazon Skills Kit (ASK) or Amazon Lex. The instructions for both are below.

Lex V1: Platform Alexa Skills Kit

Select “Amazon Skills Kit (ASK)” under “Platform” and then click “Export”.

Exporting the data in this format will give you a single .json file containing the bot name. The file may have to be reformatted to make the json display readable.

The .json file will look something like the above image. In the “intents” list, “name” refers to the intent and “samples” contains a list of the corresponding utterances.

Create a CSV file with the following columns: Botname, intent, and utterance.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

Lex V1: Platform Lex

Select “Amazon Lex” under “Platform” and then click “Export”.

Exporting the data in this format will give you a single .json file containing the bot name.

The .json file will look something like the image above. Under “resource” there is “name” which refers to the bot name. In the “intents” list there is “name” referring to the name of the intent. Under that there will be a list “sampleUtterances” which will contain the utterances of the intent.

Create a CSV file with the following columns: Botname, intent, and utterance.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

Lex V2

If your exported bot fails to parse, you can follow these manual steps to convert your Lex V2 bot to the ServisBOT common format.

From the LexV2 console, select a bot from the list, click “Actions” and from the drop-down menu select “Export”.

Exporting Lex V2 data into will give you a folder with a name including the bot name followed by the bot version, a code, and LexJson. Navigating further into the folders from the bot design export will show that each intent has its own folder and .json file.

An example of an intent json file is as follows:

“Name” refers to the name of the intent and within “sampleUtterances” there is a list of utterances belonging to the intent.

Create a CSV file with the following columns: Botname, intent, and utterance.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

Luis

If your exported bot fails to parse, you can follow these manual steps to convert your Luis bot to the ServisBOT common format.

Login to a Luis account, select a bot from the “Conversation Apps” list, and click “Export”. A dropdown menu will appear.

Select “Export as JSON” and a .json file will be downloaded with a name similar to BOTNAME.json.

The above image shows the .json file downloaded from the previous step. When navigating down to “utterances” in the file, “text” represents the utterance and “intent” represents the intent that utterance belongs to.

This data must be taken and put into a CSV file with the following columns: Botname, intent, and utterance. The Botname is the name of the bot selected from the “Conversation Apps” list.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

Watson

If your exported bot fails to parse, you can follow these manual steps to convert your Watson bot to the ServisBOT common format.

There are two ways to export Watson data and each way will result in a different format of json file. Note that bots in “Classic” do not carry over to “New” and vice versa, so bots will only exist in one of those platforms.

Your starting point, be it “Classic” or “New”, may vary. The Watson Assistant will display the current version.

Classic Experience

From the Watson Assistant, select the bot from the dropdown menu at the top of the screen. Once selected, click the icon on the top right “Manage”. Navigate to “Switch to Classic Experience”.

Once in the classic experience, select the assistant you would like to export. Click on “Skills” on the menu on the right of the screen. Click the three dots icon on the selected bot and click “Download”.

Exporting Watson data will give you a .json file with a name similar to BOTNAME-dialog.json and will look like the image below.

“Intent” will refer to the intent name while the “examples” list will hold all utterances within that intent.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

New Experience

From the Watson Assistant click “Actions” on the sidebar menu. In actions click the gear on the top right of the screen “Global Settings”.

Navigate to the “Upload/Download” tab. Click “Download”.

Exporting Watson data will give you a .json file with a name similar to BOTNAME-dialog.json and will look like the image below.

“Name” refers to the name of the bot. Intents and utterances are a little harder to locate in this json file. “Title” will refer rto an intent name and will correspond to the words and numbers listed under “intent”. In the above image “set_appointment” corresponds to “action_19432_intent_27233”.

Further down in the json file are the utterances within the intent. Notice that the utterances under “examples” belong to “action_19432_intent_27233”, which is intent “set_appointment”.

Create a CSV file with the following columns: Botname, intent, and utterance.

Each utterance should have a row in the CSV with the same Botname, and the correct intent name in the intent column, similar to the image below.

3 - EULA

THIS SERVISBOT LIMITED SOFTWARE END USER LICENSE AGREEMENT (“AGREEMENT”) IS BY AND BETWEEN SERVISBOT LIMITED, LOCATED AT 39 NORTHERUMBERLAND ROAD, BALLSBRIDGE, DUBLIN F3 H04 IRELAND (“SERVISBOT”) AND THE INDIVIDUAL OR LEGAL ENTITY WHO IS USING THE APPLICABLE SOFTWARE MADE AVAILABLE BY SERVISBOT (“CUSTOMER”) AND GOVERNS ALL USE BY CUSTOMER OF SUCH SOFTWARE. BY DOWNLOADING OR USING THE SOFTWARE YOU EXPRESSLY ACCEPT AND AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE AN INDIVIDUAL AGREEING TO THE TERMS OF THIS AGREEMENT ON BEHALF OF AN ENTITY, SUCH AS YOUR EMPLOYER, YOU REPRESENT THAT YOU HAVE THE LEGAL AUTHORITY TO BIND THAT ENTITY AND “YOU” AND “YOUR” SHALL REFER HEREIN TO SUCH ENTITY. IF YOU DO NOT HAVE SUCH AUTHORITY, OR IF YOU DO NOT AGREE WITH ALL THE TERMS OF THIS AGREEMENT, YOU MUST NOT DOWNLOAD, INSTALL, DEPLOY, OR USE THE SOFTWARE.

1. DEFINITIONS

The following capitalized terms shall have the meanings set forth below: 1.1 “Licensed Software” means the ServisBOT software licensed to you pursuant to the terms of this Agreement, excluding any Open Source Software contained therein. 1.2 “Open Source Software” means ServisBOT or third party software that is distributed or otherwise made available as “free software”, “open source software” or under a similar licensing or distribution model.

2. LICENSE

2.1 Licensed Software. Subject to your compliance with the terms and conditions of this Agreement, ServisBOT hereby grants You a limited, non-exclusive, non-transferable, non-sub-licensable license to install, copy and use the Licensed Software solely for your internal use. 2.2 Open Source Software. If applicable, Open Source Software is distributed or made available under the terms of the open source license agreements referenced in the applicable distribution or the applicable help, notices, about or source files. Copyrights and other proprietary rights to the Open Source Software are held by the copyright holders identified in the applicable distribution or the applicable help, notices, about or source files.

3. RESTRICTED ACTIVITIES

You shall not, and shall not encourage any third party to: (a) modify, adapt, alter, translate, or create derivative works of the Licensed Software; (b) reverse-engineer, decompile, disassemble, or attempt to derive the source code for the Licensed Software, in whole or in part, except to the extent that such activities are permitted under applicable law; (c) distribute, license, sublicense, lease, rent, loan, or otherwise transfer the Licensed Software to any third party; (d) remove, alter, or obscure in any way the proprietary rights notices (including copyright, patent, and trademark notices and symbols) of Docker or its suppliers contained on or within any copies of the Licensed Software; (e) use the Licensed Software for the purpose of creating a product or service competitive with the Licensed Software; (f) use the Licensed Software for any time-sharing, outsourcing, service bureau, hosting, application service provider or like purposes; (g) disclose the results of any benchmark tests on the Licensed Software without Docker’s prior written consent; or (h) use the Licensed Software other than as described in the documentation provided therewith, or for any unlawful purpose.

4. OWNERSHIP

ServisBOT and its licensors own and retain all right, title, and interest, including all intellectual property rights, in and to the Licensed Software, including any improvements, modifications, and enhancements to it. Except for the rights expressly granted in this Agreement, You shall acquire no other rights, express or implied, in or to the Licensed Software, and all rights not expressly provided to you hereunder are reserved by Docker and its licensors. All the copies of the Licensed Software provided or made available hereunder are licensed, not sold.

5. FEEDBACK

Feedback means any comments or other feedback You may provide to ServisBOT, at your sole discretion, concerning the functionality and performance of the Licensed Software, including identification of potential errors and improvements. By submitting any Feedback, You hereby assign to ServisBOT all right, title, and interest in and to the Feedback, if any.

6. NO WARRANTIES

YOU EXPRESSLY UNDERSTAND AND AGREE THAT ALL USE OF THE LICENSED SOFTWARE IS AT YOUR SOLE RISK AND THAT THE LICENSED SOFTWARE IS PROVIDED “AS IS” AND “AS AVAILABLE.” SERVISBOT, ITS SUBSIDIARIES AND AFFILIATES, AND ITS LICENSORS MAKE NO EXPRESS WARRANTIES AND DISCLAIM ALL IMPLIED WARRANTIES REGARDING THE LICENSED SOFTWARE, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT, TOGETHER WITH ANY AND ALL WARRANTIES ARISING FROM COURSE OF DEALING OR USAGE IN TRADE. NO ADVICE OR INFORMATION, WHETHER ORAL OR WRITTEN, OBTAINED FROM DOCKER OR ELSEWHERE SHALL CREATE ANY WARRANTY NOT EXPRESSLY STATED IN THIS AGREEMENT. WITHOUT LIMITING THE GENERALITY OF THE FOREGOING, SERVISBOT, ITS SUBSIDIARIES AND AFFILIATES, AND ITS LICENSORS DO NOT REPRESENT OR WARRANT TO YOU THAT: (A) YOUR USE OF THE LICENSED SOFTWARE WILL MEET YOUR REQUIREMENTS, OR (B) YOUR USE OF THE LICENSED SOFTWARE WILL BE UNINTERRUPTED, TIMELY, SECURE OR FREE FROM ERROR. NOTWITHSTANDING THE FOREGOING, NOTHING HEREIN SHALL EXCLUDE OR LIMIT SERVISBOT’S WARRANTY OR LIABILITY FOR LOSSES WHICH MAY NOT BE LAWFULLY EXCLUDED OR LIMITED BY APPLICABLE LAW. YOU UNDERSTAND AND ACKNOWLEDGE THAT THE LICENSED SOFTWARE IS NOT DESIGNED, INTENDED OR WARRANTED FOR USE IN HAZARDOUS ENVIRONMENTS REQUIRING FAIL-SAFE CONTROLS.

7. INDEMNIFICATION BY YOU

You agree to hold harmless and indemnify ServisBOT and its subsidiaries, affiliates, officers, agents, employees, advertisers, licensors, suppliers or partners from and against any third party claim arising from or in any way related to your breach of this Agreement, use of the Licensed Software, or violation of applicable laws, rules or regulations in connection with the Licensed Software, including any liability or expense arising from all claims, losses, damages (actual and consequential), suits, judgments, litigation costs and attorneys’ fees, of every kind and nature.

8. LIMITATION OF LIABILITY

YOU EXPRESSLY UNDERSTAND AND AGREE THAT SERVISBOT, ITS SUBSIDIARIES AND AFFILIATES, AND ITS LICENSORS SHALL NOT BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL CONSEQUENTIAL OR EXEMPLARY DAMAGES INCURRED BY YOU, HOWEVER CAUSED AND UNDER ANY THEORY OF LIABILITY, INCLUDING, BUT NOT LIMITED TO, ANY LOSS OF PROFIT (WHETHER INCURRED DIRECTLY OR INDIRECTLY), ANY LOSS OF GOODWILL OR BUSINESS REPUTATION, ANY LOSS OF DATA SUFFERED, COST OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, OR OTHER INTANGIBLE LOSS. THE FOREGOING LIMITATIONS ON DOCKER’S LIABILITY SHALL APPLY WHETHER OR NOT DOCKER HAS BEEN ADVISED OF OR SHOULD HAVE BEEN AWARE OF THE POSSIBILITY OF ANY SUCH LOSSES ARISING. NOTWITHSTANDING THE FOREGOING, NOTHING HEREIN SHALL EXCLUDE OR LIMIT SERVISBOT’S LIABILITY FOR LOSSES WHICH MAY NOT BE LAWFULLY EXCLUDED OR LIMITED BY APPLICABLE LAW. THE TOTAL LIABILITY OF SERVISBOT ARISING OUT OF OR RELATED TO THIS AGREEMENT WILL NOT EXCEED USD $100.

9. EXPORT RESTRICTIONS

You understand that the software provided under this Agreement may contain encryption technology and other software programs that may require an export license from the U.S. State Department and that export or re-export of the software to certain entities (such as a foreign government and its subdivisions) and certain countries is prohibited. You agree that you will comply with all applicable export and import control laws and regulations of the United States and the foreign jurisdiction in which the software is used and, in particular, You will not export or re-export the software without all required United States and foreign government licenses. You will defend, indemnify, and hold harmless ServisBOT and its suppliers and licensors from and against any violation of such laws or regulations by you or any of your agents, officers, directors or employees.

10. MISCELLANEOUS

The Licensed Software and any other software covered under this Agreement are “commercial items” as that term is defined at 48 C.F.R. 2.101; consisting of “commercial computer software” and “commercial computer software documentation” as such terms are used in 48 C.F.R. 12.212. Consistent with 48 C.F.R. 12.212 and 48 C.F.R. 227.7202-1 through 227.7202-4, all U.S. Government end users acquire the Licensed Software and any other software and documentation covered under this Agreement with only those rights set forth herein. This Agreement will be governed by the laws of the State of Delaware without reference to conflict of law principles. Each party agrees to submit to the exclusive jurisdiction of the courts located within the State of Delaware to resolve any legal matter arising from this Agreement. You may not assign any of your rights or obligations under this Agreement, whether by operation of law or otherwise, without the prior written consent of ServisBOT. Notwithstanding the foregoing, ServisBOT may assign the entirety of its rights and obligations under this Agreement without your consent. The application of the UN Convention of International Sale of Goods to this Agreement is disclaimed in its entirety. This Agreement constitutes the entire agreement between You and ServisBOT governing your use of the Licensed Software and supersedes any prior agreements between You and ServisBOT, including but not limited to, any prior versions of this Agreement. The failure of ServisBOT to enforce its rights under this Agreement at any time for any period shall not be construed as a waiver of such rights. If any provision of this Agreement is held invalid or unenforceable, the remainder of this Agreement will continue in full force and effect and the invalid or unenforceable provision shall be reformed to the extent necessary to make it valid and enforceable.