格 千格 KiloCell
Copyed to clipboard!

Robots.txt Generator

Visualise the reptile capture rules, and precisely control the access of the search engine to your website content.

The reptile access rules.

🚀 Common presets

robots.txt 














What's Robots.txt? What does it do to SEO?



            Robots.txt (robots protocol) is a plain text file placed under the root directory of the website. It tells the search engine reptiles. Googlebot, Bingbot) Which pages are accessible and which should be ignored. While it cannot force reptiles to stop capturing (which is of a recommendatory nature), mainstream search engines will strictly adhere to the agreement.
          


Core Command Detail


    

  • 
User-agent:
              Specifies the reptile name to which the rule applies.
              *
              Represent all reptiles.
            
    • 

  • 
Disallow:
              Tell the reptiles not to access the path. For example:
              /admin/
              It will stop the capture of the entire management backstage.
            
    • 

  • 
Allow:
              Clearly allows the acquisition of a child path that is taken from the parent directory Disallow.
            
    • 

  • 
Sitemap:
              The XML site map, which points to the site, helps reptiles to discover new content more efficiently.
            
    • 



Address the security risks of "sensitive pages searched"



            Many developers have forgotten to block backstages, user-owned data or temporary testing environments, leading to sensitive information in Google Searching results. By correctly Configure
            Disallow
            , you can:
          


    

  • 
Protection of privacy:
              Hide
              /api/
              、
              /config/
              or
              /tmp/
              - Waiting.
            
    • 

  • 
Save grab quota:
              Stops reptiles from wasting resources to capture duplicates of search results pages, filter pages, etc.
            
    • 

  • 
To prevent the entry of mirror stations:
              Guide reptiles to only your main domain name.
            
    • 



Common Robots.txt configuration error


The configuration error may cause the entire site to disappear from the search results. Common pits include:


    

  • 
Full station shield:
              Miswriting
              Disallow: /
              。
            
    • 

  • 
Syntax:
              Each directive shall be a single line.
            
    • 

  • 
Case sensitive:
              Path
              /Admin/
              and
              /admin/
              It's different on Linux servers.
            
    • 



Robots.txt and Security: Do not over-depend



            Remember, Robots.txt is.
            Publicly Visible
            Yeah. Do not write in it a very secret backstage path, because the assailants will also look at the document to find possible targets. For real sensitive data, use
            Organisation
            and
            No-index Meta Label
            And not just rely on Robots.txt.
          



            Using our visualizer, you can quickly build a Robots protocol that fits best practices, protect privacy while raising SEO Efficiency.