Skip to main content

​
Add URLs

The Links tab in the Document Manager lets you add web content to your knowledge base. Instead of copying and pasting from websites, BestChatBot scrapes the content for you.

​
Scraping Methods

BestChatBot offers three ways to add web content:

Direct URL

Paste a single page URL. The bot scrapes just that page.

Sitemap

Provide a sitemap URL and BestChatBot discovers all pages automatically.

Crawl

Enter a starting URL and BestChatBot crawls the site following links to discover pages.

​
How to Add URLs

1

Open Document Manager

Go to Document Manager from the dashboard sidebar and switch to the Links tab.
2

Choose Your Method

Select whether you want to add a Direct URL, Sitemap, or Crawl.
3

Enter the URL

Paste the URL into the input field. For example:
  • Direct: https://yoursite.com/faq
  • Sitemap: https://yoursite.com/sitemap.xml
  • Crawl: https://yoursite.com/docs
4

Start Scraping

Click the button to begin scraping. You’ll see progress as pages are discovered and processed.
Scraping time depends on the number of pages. A single page takes seconds; a full site crawl may take several minutes.
5

Rebuild Knowledge Base

After scraping completes, remember to Rebuild Knowledge Base so the bot can use the new content.
Links tab showing Crawl Website, Single URL, and Sitemap options with scraped URLs

​
Scraped Pages Limits

Each plan includes a set number of scraped pages:
PlanScraped Pages
Free5
Starter25
Pro50
Enterprise200
Use Direct URL for your most important pages first. Save crawling for when you need full coverage.

​
Managing Scraped Content

Scraped pages appear alongside uploaded documents in your knowledge base. You can:
  • View the URL and scrape date
  • Delete individual scraped pages
  • Re-scrape a URL to get updated content

​
Next Steps

Q&A Learning System

Learn how the bot learns from Discord conversations.

Rebuild Knowledge Base

Process your new content.