> ## Documentation Index
> Fetch the complete documentation index at: https://learn.workflow.dog/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl Website

> Crawl multiple pages from a site and return content, links, and screenshots.

**Crawl Website** starts an asynchronous Firecrawl crawl at one URL, waits for
it to finish, and returns every completed page as a list. Use it when you need
site-wide content rather than a single page.

<Note>
  The workflow remains on this action while Firecrawl is scraping. The node
  checks the crawl every two seconds and fails if Firecrawl reports a failed
  crawl.
</Note>

## Crawl inputs

| Input                       | Type              | Required | Description                                                                    |
| --------------------------- | ----------------- | -------- | ------------------------------------------------------------------------------ |
| **Third-party account**     | Firecrawl account | Yes      | The Firecrawl API key used for the crawl.                                      |
| **URL**                     | String            | Yes      | The starting URL.                                                              |
| **Limit**                   | Integer           | Yes      | Maximum pages to crawl, from `1` to `1000`. Defaults to `100`.                 |
| **Max Depth**               | Integer           | Yes      | Maximum link depth from the starting URL, from `1` to `100`. Defaults to `10`. |
| **Ignore Query Parameters** | Boolean           | No       | Treats URL variants that differ only by query parameters as the same page.     |
| **Allow Backward Links**    | Boolean           | No       | Allows the crawl to move to parent paths outside the starting URL's path.      |

## Per-page scrape inputs

These settings are applied to every page visited:

| Input                    | Type                             | Default | Description                                                     |
| ------------------------ | -------------------------------- | ------- | --------------------------------------------------------------- |
| **Format**               | `markdown`, `html`, or `rawHtml` | `html`  | Content representation for each page.                           |
| **Include Links**        | Boolean                          | Off     | Includes links found on each page.                              |
| **Include Screenshot**   | Boolean                          | Off     | Includes a screenshot for each page.                            |
| **Full Page Screenshot** | Boolean                          | Off     | Captures full-page screenshots when screenshots are enabled.    |
| **Only Main Content**    | Boolean                          | On      | Excludes headers, navigation, footers, and similar page chrome. |
| **Mobile**               | Boolean                          | Off     | Emulates a mobile device.                                       |
| **Headers**              | List of name/value pairs         | Empty   | Headers sent while scraping.                                    |
| **Wait For**             | Number                           | `0`     | Milliseconds to wait before extracting each page.               |
| **Remove Base64 Images** | Boolean                          | On      | Removes embedded base64 image data while retaining alt text.    |

## Outputs

| Output    | Type            | Description                                                                                              |
| --------- | --------------- | -------------------------------------------------------------------------------------------------------- |
| **Total** | Number          | Number of pages returned by the completed crawl.                                                         |
| **Pages** | List of objects | Crawled pages, each containing **URL**, **Content**, and any enabled **Links** or **Screenshot** values. |

Expanding **Pages** gives each page:

| Property       | Type            | Description                                     |
| -------------- | --------------- | ----------------------------------------------- |
| **URL**        | String          | Firecrawl's source URL for the page.            |
| **Content**    | String          | Content in the selected format.                 |
| **Links**      | List of strings | Present when **Include Links** is enabled.      |
| **Screenshot** | File            | Present when **Include Screenshot** is enabled. |

## Scope the crawl deliberately

Start with a modest **Limit** and **Max Depth**, inspect the returned URLs, and
increase them only when needed. Query-heavy sites can expose many near-duplicate
URLs; **Ignore Query Parameters** helps keep those from consuming the page
limit.

<Warning>
  Screenshots multiply the work and data produced by a crawl. Enable them only
  when downstream steps actually need an image for every page.
</Warning>

## Troubleshooting

<AccordionGroup>
  <Accordion title="The crawl visits too many duplicate pages">
    Enable **Ignore Query Parameters** and lower **Max Depth**. Tracking,
    filtering, and pagination parameters often create many URL variants.
  </Accordion>

  <Accordion title="Pages above the starting path are missing">
    Enable **Allow Backward Links**. Leave it off when the crawl should stay
    within a subsection of a site.
  </Accordion>

  <Accordion title="The action takes a long time">
    Reduce **Limit**, **Max Depth**, and **Wait For**. Full-page screenshots on
    every page also add substantial work.
  </Accordion>
</AccordionGroup>