Robots.txt Generator - Create Custom Robots.txt File

Search Engine Optimization Tools

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Search engines are using robots / user-agents to crawl your pages. Robots.txt is a file that contains instructions on how to crawl a website. Additionally, the robots.txt file may include links to XML-sitemaps. This is also known as the Robot Exclusion Protocol and this value is used by sites to indicate which part of their website the bots need to index. Search engines like Google use website crawlers or robots that review all the content on your website. Also, you can specify fields that you do not want to process through these crawlers. You can add these pages to the file to explicitly ignore them.

Our Robots.txt Generator tool is designed to help webmasters, SEOs and marketers create their robots.txt files without much technical knowledge. Although our tool is easy to use, we recommend that you familiarize yourself with Google's instructions before using it. This is because misapplication can cause search engines such as Google to be unable to crawl the critical pages on your site or even your entire domain, which can have a very negative effect on your SEO.

More SEO Tools:

Backlink Maker XML Sitemap Generator Htaccess Redirect Generator Domain Age Checker Plagiarism Checker