Advanced Robots TXT Generator

A/D User agent Directory or File

See how to use Advanced Robots TXT Generator

Robots Text Generator (edited by Ann Smarty) is designed to allow site owners to easily create Robots.txt files to control which sections of your site are blocked for various crawlers.

With this robots.txt generator, Google as well as several other search engines can be specified within your criteria. If you want to specify alternative directives for a crawler, just click on the User Agent list box to select a specific bot. You can either upload a file that exists in order to prepopulate the tool, or you can enter a domain in the box and click on “upload.” The tool can be used directive creation with both allow and disallow directives for User Agents for content specified on a site. Click on “Add Directive” to add a new one to the list and click on “Remove Directive” in order to edit one that exists or establish a new directive.

Customizing User Agent Directives

If you want to establish different directives for a specific crawler, click on the “User Agent list box” to pick the bot you want. You can then click on “Add Directive” and a custom section will be added to the list and will include all generic directives along with the custom directive. If you want to alter a generic Disallow directive so it is an Allow Directive, for a custom user agent, you can make a new Allow directive for the distinct user agent on the content. This will cause disallow directives that match to be removed.

If you would like some additional reading on robots.txt directives, please check out this post, The Ultimate Guide to Blocking Your Content in Search.

With this tool, you are able to include a link to your XML sitemap file by entering the URL for the file into the XML Sitemap text box. You can then click “Update” to have the command added to the file list.

When the tool is done running, the results can be exported and FTP can be used for uploading the resulting file to your site. Once this file is uploaded, search engines will know what directories or pages of your domain shouldn’t show up as results in searches.