What does the Website Crawler do?
The Website Crawler visits your website pages, extracts text content and stores it in your bot's knowledge base — all within minutes, without any manual uploads.
How it works
- Enter the start URL and optional URL include/exclude patterns.
- Set the crawl depth.
- Start the crawl — navigation and footer elements are removed, main content is retained.
- Text is automatically chunked and indexed in the vector database.
- Schedule automatic re-crawls (weekly, monthly) to keep the knowledge base current.
Limitations and ethical use
Robots.txt is respected; noindex/disallow pages are skipped. Images, videos and JS-rendered content are not extracted. The crawler may only be used on websites you own or are authorised to crawl.