Spider

Find the technical problems before Google does

The Spider visits your site page by page, as a search engine would, and hands you a score and the list of problems to fix, split by severity.

Schematic illustration, not a real screenshot.

Who it is for

  • Anyone running a site with many pages, such as shops, portals and blogs.
  • Agencies doing periodic audits who need to show improvements to clients.
  • Developers who want to check a site after a migration or a redesign.

The problem it solves

Across hundreds of pages, problems are not visible by eye

Broken links, duplicate titles, pages blocked by mistake, redirect chains: on a large site they stay invisible until traffic drops. A systematic crawl brings them to light, with the detail of where they are.

How it works

  1. Enter the starting address

    Choose the URL, how many pages to visit (200 by default) and the speed: safe, fast or turbo, with a pause between requests from 4 to 1 seconds.

  2. The crawl proceeds in a queue

    It reads robots.txt and the sitemap, follows internal links and records every page. You can follow the progress or come back later.

  3. Read the result

    Score, response codes, problems by severity and every page, with filters and search.

  4. Schedule and compare

    Plan a weekly or monthly crawl and compare two crawls to see what you fixed and what appeared.

What you get

Problems by severity

Over forty types of problem (titles and meta descriptions, H1, canonical, hreflang, images, mixed content, security headers and others), each with a severity and a weight on the score.

Page list

Every page with response code, indexability, word count and timings; filters by status and problem, search, sorting and a page detail with incoming and outgoing links.

Broken links

Internal links that lead to pages with errors and the most affected destinations.

Duplicate content

Identical titles, meta descriptions and content, with the groups of pages involved.

Sitemap viewer

The indexable pages found, with a ready-to-download sitemap.xml file.

Link graph

Most linked pages, pages with the most outgoing links and the most linked external domains.

CSV exports

Pages, problems, broken links and internal links, protected against harmful spreadsheet formulas.

Comparison and history

Score difference between two crawls of the same site, solved and new problems, new or no longer found pages and a trend chart.

Competitor Spider

Up to ten competitors to watch through a sample of public pages: duplicate titles, broken external links and changes over time.

Illustrative example

A blog after an address change

After changing its address structure, a blog runs a crawl and finds a group of internal links leading to pages with errors. It fixes them, schedules a weekly crawl and compares the last two: the solved problems show up in the list, and a new one would be seen straight away.

What it does not do and the limits to know

  • It reads the pages’ HTML as a crawler does: it does not run JavaScript, so it does not see content that only appears after dynamic loading.
  • It respects robots.txt and stops if most responses are 403 errors, so as not to overload the site: if your firewall blocks the bot, the crawl stays incomplete.
  • It visits at most the pages you set (200 by default, 10,000 the absolute limit); depth, simultaneous active crawls (3) and crawls per hour (30) have safety limits, and some depend on the plan.
  • Only public sites can be crawled: internal addresses or ones not reachable from the Internet are refused.
  • On competitors it reads only a limited sample of pages and always respects robots.txt and opt-out requests.
  • The comparison between crawls shows score, solved and new problems, new or no longer found pages, but not the detail of the problems of each single page nor competitor crawls.
  • The duplicates detected are identical content, not similar content.

Frequently asked questions

Can the Spider slow down my site?

It makes one request at a time with a pause between them (1 to 4 seconds, your choice) and respects robots.txt. On more delicate sites the safe speed is the better choice.

Can I crawl a site that is not mine?

For normal crawls we recommend doing it only on your own sites or ones you are authorised to crawl. To watch a competitor there is the Competitor Spider, which requires accepting the terms of use and respects the rules of the observed site.

What happens if the site blocks the Spider?

If most responses are 403 errors the crawl stops as «blocked». The bot has to be allowed in the site’s firewall or anti-bot service.

How can a site ask not to be watched as a competitor?

The owner can ask for exclusion: the request is verified with a message to the domain’s address, with a link valid for 48 hours, and once confirmed the site is no longer analysed.

Can I export the results?

Yes, as CSV: pages, problems, broken links and internal links. From the sitemap viewer you can download a sitemap.xml.

How often can I repeat the crawl?

Whenever you want, within the plan’s limits, or with a weekly or monthly schedule.

Ready to try it on your site?

Create the free account: analyses that have a cost always show the estimate and only start after your confirmation.