What is that?
robots.txt It is placed in the root directory of the website, which is used to alert search engines that comply with the protocol to which path to grab. It is a public document and cannot be used as a security mechanism; The real private content should use combinations of rights, privileges and noindex. This tool can be used to generate rules before accessing the website root directory to confirm that the file can return normally.
What difference does it make between robots.txt and Sitemap, noindex?
Robots.txt primarily controls whether reptiles request a path, Setmap is used to provide a list of URLs that the search engine is expected to find, and noindex is the index control signal at the page level. Disallow does not mean that the index is deleted, nor does it prevent already public sensitive URLs from being accessed; The appropriate method should be selected for the three different objectives of capture, discovery and indexing before configuration.
What does the Robots.txt generator support?
- Allows quick presets like all / ban all / standard configurations
- WordPress, Electrician, SPA Common Rule Template
- Multi User-atent, Allow/Disallow, Sitemap, Crawl-delay, Host
- Copy and Download Results
Recommended
- Make sure you get the main channel from the front desk.
- Block backstages, shopping carts, internal search loop parameter pages, etc.
- Declaration of the Sitemap address in the file
- Check root directory returns after posting and refer to Google robots.txt Official Document Certification rules
It's easier to be wrong.
Disallow: /All stations, by mistake.- Sensitivity, API, write in robots without permission.
- Rules sequence matching prefix to error of understanding
- Multi-environmental domain names are used differently
Configure Example and Release Check
The common content station allows the public page to be accessible while blocking backstage and internal search paths. For example,User-agent: * Complement Disallow: /admin/ and Disallow: /search, and add complete Sitemap: https://example.com/sitemap.xml。 The electrician website should also check shopping vans, clearing pages and screening parameters against actual URL structures, avoiding errors in the classification or commodities pages.
The file is generated to confirm that it is located in the domain name base directory, is available in plain text format, is available at the Sitemap address, and is checked for correctness when deployed in different environments. Robots.txt provides only a grab hint for standard reptiles and cannot replace login validation, server privileges or sensitive data protection.
Configure thought for different types of websites
The content stations usually simply need to keep articles, classifications and open resources accessible, while blocking backstage and internal search paths. WordPress Web site can be checked more frequently /wp-admin/, access and search parameters; The electrician website avoids blocking the commodity pages, classification pages and important filter pages. The SPA website should also confirm that the content URL, static resources and rendering portals are not inadvertently banned by the rules.
Checklist after release
- Confirms that the file is in the domain root directory and can be passed
/robots.txtBack。 - Confirms a response status of 200 and content in plain text format.
- Confirms that the Sitemap address is complete, correct and accessible.
- Checking for no error
Disallow: /, and there is no blocking of important pages. - Checks the domain name used in the test environment and the sitemap address.
Robots.txt Common Problems
Can you promise not to get caught? (b) Not be enforceable; It's a courtesy agreement, and sensitive content is subject to login and access.
Sitimap, do you want to write to robots? Recommendation to facilitate the discovery of site maps.
What happens at the station? The search engine may no longer capture the public content and be used carefully to produce the front desk.
Crawl-delay supports it? Not all reptiles support, Google usually ignores.