Skip to main content

Website Chatbot Workflow

This workflow allows you to crawl a website and turn it into a RAG-based agent. This allows you to produce a chatbot that can answer questions from your website.

For more detailed information about what this workflow is doing, see the crawl-to-rag documentation in GitHub.

Creating the workflow​

For general information on creating workflows in ApertureDB Cloud see Creating and Deleting Workflows.

Configure your workflow by selecting:

  • Which instance to use. If you only have one instance, there will be no options to select.
  • Set the URL of your website where the crawl should start. For example, https://docs.aperturedata.io/. You can include multiple URLs with spaces between them.
  • Choose an LLM provider from the list. Currently the choice is between OpenAI, Together, or Groq, but more options may be added.
  • Enter an API key for the chosen LLM provider. To acquire an LLM key, go to: OpenAI, Together, Groq. For your security, we recommend that you create a new API key rather than reusing one.
  • Optionally, if you wish to set-up monitoring using AIMon's Response monitoring service, you can supply the API key, App name and LLM name as setup in your AIMon account. If you just wish to run the workflow, you can leave these fields blank.

Website Chatbot setup screen start URLs

Once you have filled in the fields, click "Submit". Your workflow will be created and will start running.

See the results​

The API​

The RAG agent provides an API; see the documentation for more details. To connect to the API, use the URL for your server with /rag at the end. To authenticate, supply the bearer token in an authorization header:

Authorization: Bearer <token>

This token can be obtained by clicking the connection helper from the workflows section on the Website chatbot workflow.

Connection helper button

You can also explore the API using OpenAPI (sometimes referred to as "Swagger") by going to /rag/docs on the server.

The notebook rag.ipynb on jupyterlab workflow.​

In addition to the API, we have a notebook to try query this endpoint, if you've run a jupyterlab workflow.

RAG notebook in jupyterlab

This notebook contains the code to run a query against the service we stood up by running the website chatbot workflow.

One of the cells in the notebook will prompt for a Key, which is the same token that we copied for authorization.

Prompt for authorization token

After supplying that, the service could be asked all sorts of question whose answers are available in the crawled pages in the URL supplied.

RAG powered QA