What Spider Simulator does
Spider Simulator fetches one public page, follows allowed redirects, and parses the returned HTML without executing JavaScript. The report includes title, description, robots metadata, canonical URL, heading levels one through six, bounded internal and external link lists, and up to twenty thousand characters of visible text.
Real search engines use their own user agents, rendering systems, crawl schedules, indexing policies, canonicalization, and link processing. This tool does not emulate those systems or determine whether a URL is indexed. It is best used to spot what is present in a basic server-visible HTML response and what may depend on client rendering.
How to use Spider Simulator
- Enter the full public page URL you want to inspect.
- Select Simulate spider to fetch and parse one HTML response.
- Review status, metadata, heading order, links, visible text, and the truncation flag.
- Compare the result with a rendered browser, robots controls, and platform-specific inspection tools when crawl behavior matters.
Important details
No JavaScript is executed
The visible-text and link results come from the returned HTML. Client-rendered content, links added after hydration, and interactions that reveal content may be absent.
Output is intentionally bounded
Heading and link arrays have caps, and the visible-text preview is limited to twenty thousand characters. The truncated flag warns when the source contained more material than the report includes.
Internal and external links are separated
Resolved links are classified relative to the final page URL and listed as normalized URLs. The simulator inventories them but does not request each destination or judge link health.
Practical uses
- Check whether important copy and links exist in the initial HTML response.
- Review heading structure and canonical metadata during a rendering migration.
- Collect a compact server-visible page summary for technical troubleshooting.
Privacy and limitations
The URL is sent to PagesTools and fetched by its server. The destination can log the request, including any query string; avoid private or signed URLs. The JSON report uses no-store cache headers.
The simulator fetches one bounded HTML page, does not execute JavaScript, and is not any search engine's crawler. It cannot confirm indexing, rendered parity, robots.txt compliance, link health, structured data, or complete content when output is truncated.