Adding Website to DocsAI

Select Web from the available sources and proceed.

Fetch Document

Enter your website url and click Fetch document. This will list all the links that are available from the website. You can safely navigate away from this and complete the setup later after all of the pages are fetched.

Load all paths

If enabled, DocsAI will recursively crawl all links on a website until it has fetched all of them. This is recommended to ensure that no pages are missed. However, you can disable this to only extract text from the specified URL

Paths to skip

If you don't want to crawl over certain paths such as /blogs or /private use this to skip paths. The input accepts comma seperated paths.

Fetched pages

If you don't know the skip path or want to exclude some pages from the training data, you can easily make changes here. Once you're finished configuring, click "Create document" and your chatbot will be ready to answer questions from these websites.

You can also get a insight of how much data storage your website going to consume. If you have more data consider upgrading to higher plans.

Sync pages

Whenever the website gets updated , you should sync this document to get all the details updated. Let us know in the Feature Requests (opens in a new tab) if you want to automate this.