The Website Crawler of AI Assistent scans your entire website and automatically processes its content into usable knowledge for the AI. You don't need to manually retype or copy content: the crawler finds all pages on its own, reads the text, structures the information and makes it immediately available to the chatbot. Within minutes of setup, the AI knows the content of your website.
The crawler operates from a starting URL you provide and automatically follows all internal links up to a maximum depth you set yourself. You can exclude specific URL patterns — for example admin areas, shopping cart pages or other pages you don't want indexed. This gives you precise control over what the AI knows.
After the initial crawl, the system periodically checks for new or changed pages. Add a new product to your website? The crawler detects the change and updates the AI's knowledge base without you having to do anything. The chatbot always has current information, even if your website is updated frequently.
The crawler combines seamlessly with other knowledge sources. You can supplement indexed website content with manually written knowledge base articles, uploaded PDFs and product documentation. The AI combines all these sources and automatically selects the most relevant information to answer a customer question.