Skip to content

Website Loader

Warning

This feature is currently in beta testing and may not always work as intentended. While it is testing, it will only sync up to a maximum of 50 webpages, across all of the urls that you enter.

Configuration

  • Enter the name and description of your connector.

  • Enter multiple base urls into the form.

Recursive Syncing

Users should then choose the appropriate Maximum Recursion Depth - this functionally means the number of sub-links the tool will sync. By choosing 1, the tool will only sync the text on the page, and not the text of any linked pages. If you wish to sync an entire website, you may want to set this number to be high, such as 5.

Limitations

  • Not allow websites allow crawling and cannot be synced. If you encounter a website that is not loading, please let us know and we will investigate.

  • The connector will only load sites associated with the base URL that you enter in the configuration. This means that if it encounters a link to an external site, it will not ingest the contents of that site.

Tips

  • Some URLs redirect to other sites. Remember that a redirect will count as a recursion depth level when syncing. For example, if you go to www.newyorktimes.com, it will redirect to www.nytimes.com. This will count as one level if your recursive depth setting.