Website Crawler and Sitemap Builder Tool


Your privacy is important to us. We will never share your information with third parties.
If you would like to prevent this free tool from crawling your website, please add the following lines to your robots.txt file:
User-agent:NinjaBot
    Disallow: /

See how to use Website Crawler and Sitemap Builder Tool

Links are very important to websites, They allow search engine bots crawl all important pages from your site. Broken links hurt usability and prevent crawlers from accessing your deep pages.

The Google Sitemap Generator tool (by Jim Boykin) will check both internal and external links on a whole website and generate a report. The report will give insight into a website’s link structure, and will identify link errors and redirects. The results of this tool are downloadable.

Getting Started

To get started with the Google Sitemap Generator free tool, simply paste or type in site you would like scanned (you can choose to scan as little as 500 pages or as many as 10,000 pages). The tool will start immediately and run in real time. With bigger sites, the tool can take 30 minutes or more, so you have the option of having the results emailed to you once the site crawl is finished and the information is gathered. This generator also serves as an HTML sitemap generator as well as an XML sitemap generator.

Note: The results of this tool are interactive. Most information is linked and for the majority of cells that include non-linked data, you can mouse over to see all results.

Results Bar

When this tool starts running, the results bar will appear at the beginning of the page and show the following info:

  • Status of the Tool
  • Number of External Links Found
  • Number of Internal URLs Crawled
  • Number of Internal HTTP Redirects Found
  • Number of Internal HTTP Error Codes Found
  • Number of External HTTP Redirects Found
  • Number of External HTTP Error Codes Found

For those in need of sitemaps generated by either an XML or HTML sitemap generator, there are options offered here. The following are also shown:

  • Download Tool Results in Excel Format
  • Download XML Sitemap Button
  • Download Tool Results in HTML Format

Email Notification

If you would like the report to be sent through email because you do not have time to wait for it to be done crawling, you can submit your email address. This feature is offered for free to everyone.

Tables:

Tool Results

When the results are finished, the tool will organize information into six different tables:

  • External Links
  • Internal Links
  • Internal Errors
  • Internal Redirects
  • External Redirects
  • External Errors

You have the option to click on column headings to sort results and data is normally linked to details about the information or page URLs.

Internal Links

This table that is generated with the XML sitemap generator will include the following fields:

  • URLs crawled on site
  • Link to the On Page Optimization, Check Images Sizes, Alt Text and Header Check tools for the URL
  • URL level from the domain root
  • URL returned HTTP status code
  • Link text for URL
  • Number of Internal Links on Page
  • Number of External Links on Page
  • Size of the page in Kilobytes
  • Tag text from the URL’s page
  • The description tag text from the URL’s page
  • The keywords from the URL’s [age
  • Contents, if used, of the anchor tag’s “rel=” attribute

External Links

This table will include the following fields of data:

  • URL’s returned HTTP status code
  • Amount of times a URL is linked to from within the site
  • External URL used in link
  • Link text for URL
  • Internal URL on which link was first found

Internal HTTP Error Codes

This table will gather any of the pages that are returning error codes in one pace to help in organizing a way to fix the problems. This table includes the following fields:

  • URL’s returned HTTP status code
  • Number of times URL is linked to from within the site
  • Internal URL used in link
  • Link text for URL
  • Internal URL on which link was first found

Note: This table is a subcategory of the internal links table and shows just those page returning code errors.

Internal Redirects

This table gathers any pages returning redirects in one easy list so they can be reviewed. You shouldn’t have to rely on internal redirects and instead can repair the source code that contains that redirect link. The redirects table includes the following fields:

  • URL’s returned HTTP status code
  • Number of times URL is linked to from within the site
  • Internal URL used in link
  • Link text for URL
  • Redirect’s target URL
  • Internal URL on which link was first found

Note: This table is a subcategory of the internal links table and shows just pages that return 301 and 302 status code redirects.

External HTTP Error Codes

This table will gather all pages returning error codes and organize them into once page so problems can easily be resolved. This table includes the following fields:

  • URL’s returned HTTP status code
  • Number of times URL is linked to from within the site
  • Internal URL used in link
  • Link text for URL
  • Redirect’s target URL
  • Internal URL on which link was first found

Note: This table is a subcategory of the external links table and shows only pages that are returning code errors.

External Redirects

The external redirects table will combine all pages returning redirects into one uniform list so they can be easily reviewed. Since the redirect to the targeted page will not affect your page, fixing these URLs can be low on the priority list. This table includes the following fields:

  • URL’s returned HTTP status code
  • Number of times URL is linked to from within the site
  • External URL used in link
  • Link text used for URL
  • Redirect’s target URL
  • Internal URL on which link was first found

Note: This table is a subcategory of the external links table and shows only those pages that return 301 and 302 redirects.


Feedback

 

Users comments:

Login or register to post comments and rate

Not rated yet.

Anonymous
2014-11-18
great tool