Website Chatbot Workflow

This workflow allows you to crawl a website and turn it into a RAG-based agent. This allows you to produce a chatbot that can answer questions from your website.

This video demonstrates how to use this workflow to crawl your website and use it in a RAG question-answering agent.

Creating the workflow

Creating and deleting workflows

For general information on creating workflows in ApertureDB Cloud see Creating and Deleting Workflows.

1

2

3

4

5

6

7

8

1. Set the URL of your website where the crawl should start. For example, https://docs.aperturedata.io/. You can include multiple URLs with spaces between them.
2. Choose an LLM provider from the list. Currently the choice is between OpenAI, Together, or Groq, but more options may be added.
3. Enter an API key for the chosen LLM provider. To acquire an LLM key, go to: OpenAI, Together, Groq. For your security, we recommend that you create a new API key rather than reusing one.
4. Set the list of allowed CORS origin domains as a comma-separated list
5. Optionally enter an AIMon API key
6. Optionally enter an AIMon application name
7. Optionally enter an AIMon LLM model name
8. Click the blue button at the bottom

See the GitHub repository for more information

For more detailed information about what this workflow is doing, additional information about the parameters, and how to run the workflow outside of the ApertureDB Cloud, see the crawl-to-rag documentation in GitHub.

See the results

The API

The RAG agent provides an API; see the documentation for more details. To connect to the API, use the URL for your server with /rag at the end. To authenticate, supply the bearer token in an authorization header:

Authorization: Bearer <token>

This token can be obtained by clicking the connection helper from the workflows section on the Website chatbot workflow.

You can also explore the API using OpenAPI (sometimes referred to as "Swagger") by going to /rag/docs on the server.

The notebook rag.ipynb on jupyterlab workflow.

In addition to the API, we have a notebook to try query this endpoint, if you've run a jupyterlab workflow.

This notebook contains the code to run a query against the service we stood up by running the website chatbot workflow.

One of the cells in the notebook will prompt for a Key, which is the same token that we copied for authorization.

After supplying that, the service could be asked all sorts of question whose answers are available in the crawled pages in the URL supplied.