---
title: Configuring the crawler
weight: 3
---

The crawler itself can be [configured](https://github.com/spatie/crawler#usage) to do a few different things.

You can configure the crawler used by the sitemap generator, for example, to ignore robot checks:

```php
use Spatie\Crawler\Crawler;
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->configureCrawler(function (Crawler $crawler) {
        $crawler->ignoreRobots();
    })
    ->writeToFile($sitemapPath);
```

## Limiting the amount of pages crawled

You can limit the amount of pages crawled by calling `setMaximumCrawlCount`:

```php
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->setMaximumCrawlCount(500)
    ->writeToFile($sitemapPath);
```

## Setting the crawl concurrency

You can set the number of concurrent connections the crawler will use:

```php
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->setConcurrency(1)
    ->writeToFile($sitemapPath);
```

The default concurrency is 10.

## Controlling the crawl depth

You can limit how deep the crawler follows links:

```php
use Spatie\Crawler\Crawler;
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->configureCrawler(function (Crawler $crawler) {
        $crawler->depth(3);
    })
    ->writeToFile($path);
```

## Executing JavaScript

The sitemap generator can execute JavaScript on each page so it will discover links that are generated by your JS scripts. You can enable this feature by setting `execute_javascript` in the config file to `true`.

Under the hood, [headless Chrome](https://github.com/spatie/browsershot) is used to execute JavaScript. You'll need to install `spatie/browsershot` separately:

```bash
composer require spatie/browsershot
```

Here are some pointers on [how to install it on your system](https://spatie.be/docs/browsershot/v4/requirements).

The package will make an educated guess as to where Chrome is installed on your system. You can also set the path in `config/sitemap.php`.