A site crawl follows a set of discovery rules to inspect multiple public pages and build evidence about their relationships. It can reveal internal links, response problems, redirects and repeated page signals. Its usefulness depends on understanding the starting point, boundaries and collection limits. A crawl result is an observed graph, not automatic proof that every document owned by an organization was found.
Before starting, define the website area and the questions you want to investigate. A navigation review, a migration check and a content inventory may need different inputs. Identify the public origin, representative entry pages and any publishing inventory available for comparison. This makes it easier to interpret both discovered pages and important pages the crawl did not reach.
Choose a meaningful entry point
Start with a public URL that connects to the area you intend to examine. A homepage is useful for general navigation, but a deeply separated documentation area may need its own appropriate entry. Record the initial address and any redirect that changes where discovery begins. A request ending on another host or a generic screen changes what the crawl can meaningfully observe.
Open the site crawler and review the scope shown by the workflow. Use public addresses without credentials or private tokens. If the starting request fails, investigate the response evidence before treating the result as an evaluation of content the crawler never retrieved.
Understand the boundaries of the discovered graph
A crawler follows links it can extract within the configured scope and budgets. Pages requiring authentication, hidden behind unsupported interactions or absent from the discovered link graph may remain unvisited. A maximum page count or time budget can also stop collection while additional URLs still exist. Keep those limits with the export and final explanation.
Do not describe an unvisited URL as healthy merely because no failure was reported for it. Likewise, a page absent from a bounded crawl is not automatically an orphan. Compare the observed graph with the publishing inventory, another relevant entry point or the intended sitemap. The sitemap explanation helps distinguish a declared URL list from actual link discovery.
Inspect page relationships as well as page totals
Record which pages link to important destinations and how a visitor reaches them from relevant starting points. A large number of discovered pages does not necessarily indicate good architecture. A small group of useful category pages may provide clearer routes than thousands of generated filter combinations with little meaningful distinction.
For a fictional tutorial library, an advanced setup article might be linked only from an obsolete release note. The practical question is whether current readers have an understandable path to it. Add or update relevant contextual navigation when appropriate, rather than attaching the article indiscriminately to every page. The heading article discusses making the destination itself easier to navigate once it is reached.
Follow failures back to their source
When a crawl finds a failed destination, retain the source page and link. Several failures may originate in one shared menu or template. Group the underlying problem while keeping representative examples for verification. This helps the person responsible fix the cause instead of treating each repeated instance as unrelated work.
Use the broken-link workflow for that source-to-target investigation. Distinguish network failures, restricted requests, error responses and intentional redirects. An automated-request challenge does not establish that a resource is permanently dead, and a redirect reaching a successful response does not establish that the destination is relevant to the original link.
Reconcile URL identity across the sample
Compare canonical declarations, redirect destinations and sitemap entries for the pages you inspect. Unexpected disagreement may reveal inconsistent URL construction or a publishing rule applied to the wrong template. Review actual documents rather than deciding that every different URL must be a duplicate or every canonical target must be correct.
The canonical guide and indexability article explain those separate decisions. A crawl can expose patterns worth investigating, but a collection of similarity signals does not replace an editorial comparison of the content and purpose. Keep the chosen preferred-URL policy documented so future releases do not reintroduce contradictory variants.
Account for rendering and crawler access
Some sites expose most links in the initial HTML; others create parts of the document after scripts run. Record whether a page was measured from the raw response or with rendered evidence. Rendering availability and budgets can affect coverage. Do not describe an HTTP-only collection as a complete test of interactions it never executed.
Review relevant robots rules and any access or rate-control response encountered during collection. The robots.txt article explains why crawl preferences are distinct from privacy and indexing decisions. A public crawl should stay within its supported scope and should not attempt to bypass legitimate access controls to inflate its page count.
Use focused checks to investigate a page
A sitewide observation and a complete page-check report serve different purposes. When a crawl identifies an important example, open the relevant focused tool or complete audit for more detail on that submitted page. Do not imply that every page discovered by the crawler received every entry in the page-check registry.
Preserve the page URL and collection time when moving between workflows. The website may change between requests, and a different final destination can alter the evidence. An understandable report explains which workflow measured which observation rather than combining results into an unsupported claim about universal coverage.
Review the graph again after repairs
After changing navigation or routing, repeat the original crawl scope and inspect the affected routes directly. Check the source links and intended final pages. Compare coverage changes with the actual publishing decision: more discovered URLs may reflect a repaired link, while fewer may reflect a deliberately removed generated route rather than lost useful content.
Save the relevant exports and use the repeat-scan method for page-level before-and-after evidence where needed. A good crawl review leaves the owner with a clear picture of the observed public graph, a prioritized set of source changes and an explicit account of what was not measured.
Compare discovery and single-page inspection
| Approach | Evidence | Useful for |
|---|---|---|
| Site crawler | Relationships across discovered public pages | Reviewing the observed navigation graph |
| Complete page audit | Available registry results for one submitted page | Investigating a representative document deeply |
| Publishing inventory | The intended public content list | Finding omissions beyond a bounded crawl |
Using CrawlNFix for this task
The CrawlNFix crawler is free to use without a public-site account or paid plan. It needs an online connection to discover and request pages. Download the collected evidence for an offline architecture discussion, then run another connected crawl when you need current results. Local saved-content comparison is available for a different task and should not be presented as an offline crawl of websites that were never fetched.