Crawl Website

Crawl Website starts an asynchronous Firecrawl crawl at one URL, waits for it to finish, and returns every completed page as a list. Use it when you need site-wide content rather than a single page.

The workflow remains on this action while Firecrawl is scraping. The node checks the crawl every two seconds and fails if Firecrawl reports a failed crawl.

Crawl inputs

Input	Type	Required	Description
Third-party account	Firecrawl account	Yes	The Firecrawl API key used for the crawl.
URL	String	Yes	The starting URL.
Limit	Integer	Yes	Maximum pages to crawl, from `1` to `1000`. Defaults to `100`.
Max Depth	Integer	Yes	Maximum link depth from the starting URL, from `1` to `100`. Defaults to `10`.
Ignore Query Parameters	Boolean	No	Treats URL variants that differ only by query parameters as the same page.
Allow Backward Links	Boolean	No	Allows the crawl to move to parent paths outside the starting URL’s path.

Per-page scrape inputs

These settings are applied to every page visited:

Input	Type	Default	Description
Format	`markdown`, `html`, or `rawHtml`	`html`	Content representation for each page.
Include Links	Boolean	Off	Includes links found on each page.
Include Screenshot	Boolean	Off	Includes a screenshot for each page.
Full Page Screenshot	Boolean	Off	Captures full-page screenshots when screenshots are enabled.
Only Main Content	Boolean	On	Excludes headers, navigation, footers, and similar page chrome.
Mobile	Boolean	Off	Emulates a mobile device.
Headers	List of name/value pairs	Empty	Headers sent while scraping.
Wait For	Number	`0`	Milliseconds to wait before extracting each page.
Remove Base64 Images	Boolean	On	Removes embedded base64 image data while retaining alt text.

Outputs

Output	Type	Description
Total	Number	Number of pages returned by the completed crawl.
Pages	List of objects	Crawled pages, each containing URL, Content, and any enabled Links or Screenshot values.

Expanding Pages gives each page:

Property	Type	Description
URL	String	Firecrawl’s source URL for the page.
Content	String	Content in the selected format.
Links	List of strings	Present when Include Links is enabled.
Screenshot	File	Present when Include Screenshot is enabled.

Scope the crawl deliberately

Start with a modest Limit and Max Depth, inspect the returned URLs, and increase them only when needed. Query-heavy sites can expose many near-duplicate URLs; Ignore Query Parameters helps keep those from consuming the page limit.

Screenshots multiply the work and data produced by a crawl. Enable them only when downstream steps actually need an image for every page.

Troubleshooting

The crawl visits too many duplicate pages

Enable Ignore Query Parameters and lower Max Depth. Tracking, filtering, and pagination parameters often create many URL variants.

Pages above the starting path are missing

Enable Allow Backward Links. Leave it off when the crawl should stay within a subsection of a site.

The action takes a long time

Reduce Limit, Max Depth, and Wait For. Full-page screenshots on every page also add substantial work.

Overview

Fundamentals

Web, Files & Data

AI

Scraping & Rendering

Google Workspace

Email, CRM & Scheduling

Databases & Tables

Microsoft

Crawl inputs

Per-page scrape inputs

Outputs

Scope the crawl deliberately

Troubleshooting

​Crawl inputs

​Per-page scrape inputs

​Outputs

​Scope the crawl deliberately

​Troubleshooting

Crawl inputs

Per-page scrape inputs

Outputs

Scope the crawl deliberately

Troubleshooting