A Website represents one site and its crawl configuration. Each accepted page becomes a pipeline-owned Web page Source beneath it. Configure the Website instead of adding individual URLs as unrelated authored items.
- 1
Add the start URL
Use the canonical public address for the part of the site you want Tindre to follow.
- 2
Set boundaries
Include or exclude paths so navigation, archives, and irrelevant sections do not flood the library.
- 3
Start the crawl
Tindre discovers pages, extracts useful content, and creates child Sources.
- 4
Review the result
Check page count, failures, and examples before attaching the Website to an Agent.
Scope before crawling
Begin with the smallest path that contains the useful material. Exclude search pages, login areas, calendars, duplicate language variants, and generated archives unless they are intentionally part of the answer scope.
Understand the children
Web page Sources inherit their relationship to the Website. They are refreshed and removed by crawl reconciliation. Their extracted content is not an independent document for editors to rewrite.
Recrawl after meaningful changes
Run another crawl when the site structure or source content changes. A successful fetch does not guarantee that the page contains useful main content, so review representative results.
Handle failures narrowly
Authentication, blocked browser access, redirects, and extraction rules can fail different pages for different reasons. Adjust the relevant Website setting instead of widening every boundary at once.