Adding Website to DocsAI
Select Web from the available sources and proceed.
Enter your website url and click
Fetch document. This will list all the links that are available from the website. You can safely navigate away from this and complete the setup later after all of the pages are fetched.
If enabled, DocsAI will recursively crawl all links on a website until it has fetched all of them. This is recommended to ensure that no pages are missed. However, you can disable this to only extract text from the specified URL
If you don't want to crawl over certain paths such as
/private use this to skip paths. The input accepts comma seperated paths.
If you don't know the skip path or want to exclude some pages from the training data, you can easily make changes here. Once you're finished configuring, click "Create document" and your chatbot will be ready to answer questions from these websites.
You can also get a insight of how much data storage your website going to consume. If you have more data consider upgrading to higher plans.
Whenever the website gets updated , you should sync this document to get all the details updated. Let us know in the Feature Requests (opens in a new tab) if you want to automate this.