What a robots.txt file controls
A robots.txt file gives automated crawlers instructions about which URL paths they may request. The file normally lives at the root of a host, such as example.com/robots.txt, and groups directives under one or more user-agent lines. A rule can address a named crawler or use an asterisk for a general group. Allow and disallow directives then describe path prefixes that the group may or may not crawl.
Robots directives manage crawling, not access control. A blocked URL can still be visited by a person, requested by software that ignores the standard, or appear in search results when another page links to it. Password protection, application authorization, and server permissions are the appropriate controls for confidential material. Never rely on robots.txt to hide customer records, private documents, staging credentials, or administration endpoints.
How to create useful crawler rules
Select a user agent and add only the paths that need a deliberate rule. A slash represents the whole host, while a more specific path limits the instruction to that section. Use an allow rule when a permitted child path sits inside a broader blocked path. Keep paths relative to the host and preserve meaningful capitalization because some servers treat upper- and lower-case URLs as different resources.
Add a fully qualified sitemap URL when the site publishes an XML sitemap. Multiple sitemap directives are possible when the site uses a sitemap index or maintains separate collections. Crawl-delay is recognized by some crawlers but is not part of a universally implemented core, so it should not be treated as a dependable rate limit. Server-side throttling and caching provide stronger protection against excessive traffic.
A short file is easier to audit than a long list of generated patterns. Group rules by crawler, remove duplicates, and add a restrictive directive only when its effect is understood. Blocking scripts, styles, or other assets required to render a page can make the page harder for a search engine to evaluate. Broad wildcard patterns may also match more URLs than their author intended.
Publishing and checking the generated file
Download or copy the output as plain UTF-8 text named robots.txt, then publish it at the root of the exact protocol and host it governs. A file on www.example.com does not automatically govern a different subdomain. Request the public URL in a private browser window and confirm that it returns the intended text with a successful status rather than an HTML error page, login form, or redirect loop.
Test representative allowed and blocked URLs with the diagnostic tools offered by the search engines that matter to the site. Review the file again after a migration, domain change, content-management update, or staging deployment. A common launch error is copying a staging rule that disallows the entire site into production. Monitoring crawl reports and server logs can reveal unexpected changes, although logs do not prove that every crawler interpreted every directive identically.
Important limits of the robots standard
Crawler support differs. Major search engines publish their own interpretation details, and less established bots may ignore the file completely. Pattern matching, end-of-string markers, precedence, comments, and crawl-delay can vary. This generator creates conventional syntax; it does not contact a crawler, inspect a deployed site, validate a sitemap, or guarantee indexing and ranking outcomes.
Disallowing a page does not necessarily remove an already indexed URL. Search removal may require a noindex response that a crawler can actually fetch, authenticated removal tooling, or permanent deletion with an appropriate HTTP status. Combining a disallow rule with an unseen noindex directive can prevent the crawler from observing the directive. Plan deindexing separately from crawl-budget management.
Local processing, privacy, and review
The form assembles the robots.txt text locally in your browser. PagesTools does not need to receive the host, sitemap address, user agents, or paths you enter. Even so, a robots file becomes public after publication and can advertise otherwise obscure paths. Avoid descriptive private directory names and secure sensitive locations independently before publishing any rule.
Treat the generated text as a draft for an informed human review. Check it against current crawler documentation, the real URL structure, and the site's release process. Save the approved version in source control when practical so later changes can be compared and reversed. Recheck the live response after deployment because a correct draft can still be altered by hosting, routing, caching, or encoding configuration.