# Advanced Scraping Guide Source: https://docs.firecrawl.dev/advanced-scraping-guide Configure scrape options, browser actions, crawl, map, and the agent endpoint with Firecrawl's full API surface. Reference for every option across Firecrawl's scrape, crawl, map, and agent endpoints. ## Basic scraping To scrape a single page and get clean markdown content, use the `/scrape` endpoint. ```python Python theme={null} # pip install firecrawl from firecrawl import Firecrawl firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY") doc = firecrawl.scrape("https://firecrawl.dev") print(doc.markdown) ``` ```js Node theme={null} // npm install firecrawl import { Firecrawl } from 'firecrawl-js'; const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' }); const doc = await firecrawl.scrape('https://firecrawl.dev'); console.log(doc.markdown); ``` ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://docs.firecrawl.dev" }' ``` ## Scraping PDFs Firecrawl supports PDFs. Use the `parsers` option (e.g., `parsers: ["pdf"]`) when you want to ensure PDF parsing. You can control the parsing strategy with the `mode` option: * **`auto`** (default) — attempts fast text-based extraction first, then falls back to OCR if needed. * **`fast`** — text-based parsing only (embedded text). Fastest, but skips scanned/image-heavy pages. * **`ocr`** — forces OCR parsing on every page. Use for scanned documents or when `auto` misclassifies a page. `{ type: "pdf" }` and `"pdf"` both default to `mode: "auto"`. ## Scraping images Firecrawl also OCRs raster images (PNG, JPEG, JPEG 2000, TIFF, GIF, BMP, WebP, AVIF) and returns the text as markdown. SVG and HEIC are not supported. The `image` parser is on by default, so you do not need to set it. It has no options: `"image"` and `{ "type": "image" }` are the same. The default is `parsers: ["pdf", "image"]`. If you pass a list, parsers that are not in the list are off. For example, `parsers: ["pdf"]` turns off image OCR, so an image URL fails as an unsupported file. ```json theme={null} "parsers": [{ "type": "pdf", "mode": "fast", "maxPages": 50 }] ``` ## Scrape options When using the `/scrape` endpoint, you can customize the request with the following options. ### Formats (`formats`) The `formats` array controls which output types the scraper returns. Default: `["markdown"]`. **String formats**: pass the name directly (e.g. `"markdown"`). | Format | Description | | - | - | | `markdown` | Page content converted to clean Markdown. | | `html` | Processed HTML with unnecessary elements removed. | | `rawHtml` | Original HTML exactly as returned by the server. | | `rawBase64` | Base64-encoded original HTTP response body, as a bare Base64 string. Must be the only format in the request. The MIME type is in `metadata.contentType`. | | `links` | All links found on the page. | | `images` | All images found on the page. | | `summary` | An LLM-generated summary of the page content. | | `branding` | Extracts brand identity (colors, fonts, typography, spacing, UI components). | | `product` | Extracts a structured product (title, price, availability, images, variants) from product pages via multi-source structured data. | **Object formats**: pass an object with `type` and additional options. | Format | Options | Description | | - | - | - | | `json` | `prompt?: string`, `schema?: object` | Extract structured data using an LLM. Provide a JSON schema and/or a natural-language prompt (max 10,000 characters). | | `screenshot` | `fullPage?: boolean`, `quality?: number`, `viewport?: { width, height }` | Capture a screenshot. Max one per request. Viewport max resolution is 7680×4320. Screenshot URLs expire after 24 hours. | | `changeTracking` | `modes?: ("json" \| "git-diff")[]`, `tag?: string`, `schema?: object`, `prompt?: string` | Track changes between scrapes. Requires `"markdown"` to also be in the formats array. | | `attributes` | `selectors: [{ selector: string, attribute: string }]` | Extract specific HTML attributes from elements matching CSS selectors. | ### Mobile scraping Set `mobile: true` to emulate a mobile device. This is useful when a responsive site hides content on desktop or serves a different layout to mobile browsers. For region-specific sites, combine with `location` and a mobile screenshot to verify the rendered layout: ```python Python theme={null} from firecrawl import Firecrawl firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY") doc = firecrawl.scrape( "https://example.com", mobile=True, location={"country": "GB", "languages": ["en-GB"]}, formats=[ "markdown", {"type": "screenshot", "fullPage": True, "viewport": {"width": 390, "height": 844}}, ], only_main_content=False, wait_for=2000, ) print(doc.markdown) ``` ```js Node theme={null} import { Firecrawl } from 'firecrawl-js'; const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' }); const doc = await firecrawl.scrape('https://example.com', { mobile: true, location: { country: 'GB', languages: ['en-GB'] }, formats: [ 'markdown', { type: 'screenshot', fullPage: true, viewport: { width: 390, height: 844 } }, ], onlyMainContent: false, waitFor: 2000, }); console.log(doc.markdown); ``` ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://example.com", "mobile": true, "location": { "country": "GB", "languages": ["en-GB"] }, "formats": [ "markdown", { "type": "screenshot", "fullPage": true, "viewport": { "width": 390, "height": 844 } } ], "onlyMainContent": false, "waitFor": 2000 }' ``` If the site still serves a desktop layout despite `mobile: true`, add a mobile User-Agent via `headers`: ```json theme={null} { "headers": { "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1" } } ``` ### Content filtering These parameters control which parts of the page appear in the output. When `onlyMainContent` is `true` (the default), boilerplate (nav, footer, etc.) is stripped. `includeTags` and `excludeTags` are applied against the original page DOM, not the post-filtered result, so your selectors should target elements as they appear in the source HTML. Set `onlyMainContent: false` to use the full page as the starting point for tag filtering. | Parameter | Type | Default | Description | | - | - | - | - | | `onlyMainContent` | `boolean` | `true` | Return only the main content. Set `false` for the full page. | | `includeTags` | `array` | — | CSS selectors to include — tags, classes, IDs, or attribute selectors (e.g. `["h1", "p", ".main-content", "[data-testid=\"main\"]"]`). | | `excludeTags` | `array` | — | CSS selectors to exclude — tags, classes, IDs, or attribute selectors (e.g. `["#ad", "#footer", "[role=\"banner\"]"]`). | ### Timing and cache | Parameter | Type | Default | Description | | - | - | - | - | | `waitFor` | `integer` (ms) | `0` | Extra wait time before scraping, on top of smart-wait. Use sparingly. | | `maxAge` | `integer` (ms) | `172800000` | Return a cached version if fresher than this value (default is 2 days). Set `0` to always fetch fresh. | | `timeout` | `integer` (ms) | `60000` | Max request duration before aborting (default is 60 seconds). Minimum is 1000 (1 second). | ### Prompt injection check | Parameter | Type | Default | Description | | - | - | - | - | | `checkPromptInjection` | `boolean` | `false` | Scan the page content for prompt injection attempts before any LLM-backed format (such as `json`, `summary` or `question`) runs. Works with any format except `rawBase64`. A detection fails the request with a `403` and `SCRAPE_PROMPT_INJECTION_DETECTED`. Adds 4 credits when the check scans the whole page. See [Prompt injection detection](/features/llm-extract#prompt-injection-detection). | ### PDF parsing | Parameter | Type | Default | Description | | - | - | - | - | | `parsers` | `array` | `["pdf", "image"]` | Controls PDF and image processing. Each item is `"pdf"`, `"image"`, or a parser object. A list turns off the parsers it leaves out. `[]` to skip parsing and return the PDF as base64 (1 credit flat). | ```json theme={null} { "type": "pdf", "mode": "fast" | "auto" | "ocr", "maxPages": 10, "pages": true, "blocks": true, "pageMarkers": true } ``` | Property | Type | Default | Description | | - | - | - | - | | `type` | `"pdf"` | *(required)* | Parser type. | | `mode` | `"fast" \| "auto" \| "ocr"` | `"auto"` | `fast`: text-based extraction only. `auto`: fast with OCR fallback. `ocr`: force OCR. | | `maxPages` | `integer` | — | Cap the number of pages to parse. | | `pages` | `boolean` | `false` | Also return physical per-page markdown in the document's `pages` field. No additional cost. | | `blocks` | `boolean` | `false` | Also return per-page typed layout blocks (normalized bounding boxes, block types, reading order, markdown character spans) in the document's `blocks` field. No additional cost. | | `pageMarkers` | `boolean` | `false` | Annotate page breaks in the document markdown with `` markers (between pages only; numbering may skip pages merged across a break — see [Parse](/features/parse#page-markers-pdf)). No additional cost. | The `image` parser object is `{ "type": "image" }`. It has no other properties, and the PDF properties above do not apply to images. ### Actions Run browser actions before scraping. This is useful for dynamic content, navigation, or user-gated pages. You can include up to 50 actions per request, and the combined wait time across all `wait` actions and `waitFor` must not exceed 60 seconds. | Action | Parameters | Description | | - | - | - | | `wait` | `milliseconds?: number`, `selector?: string` | Wait for a fixed duration **or** until an element is visible (provide one, not both). When using `selector`, times out after 30 seconds. | | `click` | `selector: string`, `all?: boolean` | Click an element matching the CSS selector. Set `all: true` to click every match. | | `write` | `text: string` | Type text into the currently focused field. You must focus the element with a `click` action first. | | `press` | `key: string` | Press a keyboard key (e.g. `"Enter"`, `"Tab"`, `"Escape"`). | | `scroll` | `direction?: "up" \| "down"`, `selector?: string` | Scroll the page or a specific element. Direction defaults to `"down"`. | | `screenshot` | `fullPage?: boolean`, `quality?: number`, `viewport?: { width, height }` | Capture a screenshot. Max viewport resolution is 7680×4320. | | `scrape` | *(none)* | Capture the current page HTML at this point in the action sequence. | | `executeJavascript` | `script: string` | Run JavaScript code in the page. Return values are available in the `actions.javascriptReturns` array of the response. | | `pdf` | `format?: string`, `landscape?: boolean`, `scale?: number` | Generate a PDF. Supported formats: `"A0"` through `"A6"`, `"Letter"`, `"Legal"`, `"Tabloid"`, `"Ledger"`. Defaults to `"Letter"`. | ```python Python theme={null} from firecrawl import Firecrawl firecrawl = Firecrawl(api_key='fc-YOUR-API-KEY') doc = firecrawl.scrape('https://example.com', { 'actions': [ { 'type': 'wait', 'milliseconds': 1000 }, { 'type': 'click', 'selector': '#accept' }, { 'type': 'scroll', 'direction': 'down' }, { 'type': 'click', 'selector': '#q' }, { 'type': 'write', 'text': 'firecrawl' }, { 'type': 'press', 'key': 'Enter' }, { 'type': 'wait', 'milliseconds': 2000 } ], 'formats': ['markdown'] }) print(doc.markdown) ``` ```js Node theme={null} import { Firecrawl } from 'firecrawl-js'; const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' }); const doc = await firecrawl.scrape('https://example.com', { actions: [ { type: 'wait', milliseconds: 1000 }, { type: 'click', selector: '#accept' }, { type: 'scroll', direction: 'down' }, { type: 'click', selector: '#q' }, { type: 'write', text: 'firecrawl' }, { type: 'press', key: 'Enter' }, { type: 'wait', milliseconds: 2000 } ], formats: ['markdown'] }); console.log(doc.markdown); ``` ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://example.com", "actions": [ { "type": "wait", "milliseconds": 1000 }, { "type": "click", "selector": "#accept" }, { "type": "scroll", "direction": "down" }, { "type": "click", "selector": "#q" }, { "type": "write", "text": "firecrawl" }, { "type": "press", "key": "Enter" }, { "type": "wait", "milliseconds": 2000 } ], "formats": ["markdown"] }' ``` #### Action execution notes * **Write** requires a preceding `click` to focus the target element. * **Scroll** accepts an optional `selector` to scroll a specific element instead of the page. * **Wait** accepts either `milliseconds` (fixed delay) or `selector` (wait until visible). * Actions run **sequentially**: each step completes before the next begins. * Actions are **not supported for PDFs**. If the URL resolves to a PDF the request will fail. #### Advanced action examples **Taking a screenshot:** ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://example.com", "actions": [ { "type": "click", "selector": "#load-more" }, { "type": "wait", "milliseconds": 1000 }, { "type": "screenshot", "fullPage": true, "quality": 80 } ] }' ``` **Clicking multiple elements:** ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://example.com", "actions": [ { "type": "click", "selector": ".expand-button", "all": true }, { "type": "wait", "milliseconds": 500 } ], "formats": ["markdown"] }' ``` **Generating a PDF:** ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://example.com", "actions": [ { "type": "pdf", "format": "A4", "landscape": false } ] }' ``` **Executing JavaScript (e.g. extracting embedded page data):** ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://example.com", "actions": [ { "type": "executeJavascript", "script": "document.querySelector(\"#__NEXT_DATA__\").textContent" } ], "formats": ["markdown"] }' ``` The return value of each `executeJavascript` action is captured in the `actions.javascriptReturns` array of the response. ### Full scrape example The following request combines multiple scrape options: ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://docs.firecrawl.dev", "formats": [ "markdown", "links", "html", "rawHtml", { "type": "screenshot", "fullPage": true, "quality": 80 } ], "includeTags": ["h1", "p", "a", ".main-content"], "excludeTags": ["#ad", "#footer"], "onlyMainContent": false, "waitFor": 1000, "timeout": 15000, "parsers": ["pdf"] }' ``` This request returns markdown, HTML, raw HTML, links, and a full-page screenshot. It scopes content to `

`, `

`, ``, and `.main-content` while excluding `#ad` and `#footer`, waits 1 second before scraping, sets a 15 second timeout, and enables PDF parsing. See the full [Scrape API reference](https://docs.firecrawl.dev/api-reference/endpoint/scrape) for details. ## JSON extraction via formats Use the JSON format object in `formats` to extract structured data in one pass: ```python Python theme={null} from firecrawl import Firecrawl firecrawl = Firecrawl(api_key='fc-YOUR-API-KEY') doc = firecrawl.scrape('https://firecrawl.dev', { 'formats': [{ 'type': 'json', 'prompt': 'Extract the features of the product', 'schema': { 'type': 'object', 'properties': { 'features': { 'type': 'object' } }, 'required': ['features'] } }] }) print(doc.json) ``` ```js Node theme={null} import { Firecrawl } from 'firecrawl-js'; const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' }); const doc = await firecrawl.scrape('https://firecrawl.dev', { formats: [{ type: 'json', prompt: 'Extract the features of the product', schema: { type: 'object', properties: { features: { type: 'object' } }, required: ['features'] } }] }); console.log(doc.json); ``` ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://firecrawl.dev", "formats": [{ "type": "json", "prompt": "Extract the features of the product", "schema": {"type": "object", "properties": {"features": {"type": "object"}}, "required": ["features"]} }] }' ``` ## Agent endpoint Use the `/v2/agent` endpoint for autonomous, multi-page data extraction. The agent runs asynchronously: you start a job, then poll for results. [Agent](/features/agent) is the canonical reference for this endpoint, including execution traces, webhooks, and the full parameter list. ### Agent options | Parameter | Type | Default | Description | | - | - | - | - | | `prompt` | `string` | *(required)* | Natural-language instructions describing what data to extract (max 10,000 characters). | | `urls` | `array` | — | URLs to constrain the agent to. | | `schema` | `object` | — | JSON schema to structure the extracted data. | | `maxCredits` | `number` | `2500` | Maximum credits the agent can spend. The dashboard supports up to 2,500; for higher limits, set this via the API (values above 2,500 are always billed as paid requests). | | `strictConstrainToURLs` | `boolean` | `false` | When `true`, the agent only visits the provided URLs. | | `model` | `string` | `"spark-2"` | AI model to use. Spark 1 models are deprecated and currently route to `"spark-2"`. | | `effort` | `string` | *(unset)* | Reasoning budget: `"low"`, `"medium"`, or `"high"`. Every run executes on `"spark-2"`, so you can send `effort` with or without `model`. | ```python Python theme={null} from firecrawl import Firecrawl firecrawl = Firecrawl(api_key='fc-YOUR-API-KEY') # Start agent job started = firecrawl.start_agent( prompt="Extract the title and description", urls=["https://docs.firecrawl.dev"], schema={"type": "object", "properties": {"title": {"type": "string"}, "description": {"type": "string"}}, "required": ["title"]} ) # Poll status status = firecrawl.get_agent_status(started["id"]) print(status.get("status"), status.get("data")) ``` ```js Node theme={null} import { Firecrawl } from 'firecrawl'; const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' }); // Start agent job const started = await firecrawl.startAgent({ prompt: 'Extract the title and description', urls: ['https://docs.firecrawl.dev'], schema: { type: 'object', properties: { title: { type: 'string' }, description: { type: 'string' } }, required: ['title'] } }); // Poll status const status = await firecrawl.getAgentStatus(started.id); console.log(status.status, status.data); ``` ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/agent \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "prompt": "Extract the title and description", "urls": ["https://docs.firecrawl.dev"], "schema": {"type": "object", "properties": {"title": {"type": "string"}, "description": {"type": "string"}}, "required": ["title"]} }' ``` ### Check agent status Poll `GET /v2/agent/{jobId}` to check progress. The response `status` field will be `"processing"`, `"completed"`, or `"failed"`. ```bash cURL theme={null} curl -X GET https://api.firecrawl.dev/v2/agent/YOUR-JOB-ID \ -H 'Authorization: Bearer fc-YOUR-API-KEY' ``` The Python and Node SDKs also provide a convenience method (`firecrawl.agent()`) that starts the job and polls automatically until completion. ## Crawling multiple pages To crawl multiple pages, use the `/v2/crawl` endpoint. The crawl runs asynchronously and returns a job ID. Use the `limit` parameter to control how many pages are crawled. If omitted, the crawl will process up to 10,000 pages. ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/crawl \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://docs.firecrawl.dev", "limit": 10 }' ``` ### Response ```json theme={null} { "id": "1234-5678-9101" } ``` ### Check crawl job Use the job ID to check the status of a crawl and retrieve its results. ```bash cURL theme={null} curl -X GET https://api.firecrawl.dev/v2/crawl/1234-5678-9101 \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' ``` If the content is larger than 10MB or the crawl job is still running, the response may include a `next` parameter, a URL to the next page of results. ### Crawl prompt and params preview You can provide a natural-language `prompt` to let Firecrawl derive crawl settings. Preview them first: ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/crawl/params-preview \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://docs.firecrawl.dev", "prompt": "Extract docs and blog" }' ``` ### Crawler options When using the `/v2/crawl` endpoint, you can customize crawling behavior with the following options. #### Path filtering | Parameter | Type | Default | Description | | - | - | - | - | | `includePaths` | `array` | — | Regex patterns for URLs to include (pathname only by default). | | `excludePaths` | `array` | — | Regex patterns for URLs to exclude (pathname only by default). | | `regexOnFullURL` | `boolean` | `false` | Match patterns against the full URL instead of just the pathname. | > [!WARNING] > > The starting URL is also checked against `includePaths`. If it does not match any of the patterns, the crawl may return 0 pages. Patterns use Rust regex (RE2-style) syntax, so look-around and backreferences are not supported; a pattern that does not compile is rejected with a `400`. Each field accepts at most 1000 patterns of at most 2000 characters each, and `includePaths` and `excludePaths` together may contain at most 1000 patterns and 100,000 characters. For keyword-based filtering, send one short pattern per term instead of combining terms into a single long alternation. #### Crawl scope | Parameter | Type | Default | Description | | - | - | - | - | | `maxDiscoveryDepth` | `integer` | — | Max link-depth for discovering new URLs. | | `limit` | `integer` | `10000` | Max pages to crawl. | | `crawlEntireDomain` | `boolean` | `false` | Explore siblings and parents to cover the entire domain. | | `allowExternalLinks` | `boolean` | `false` | Follow links to external domains. | | `allowSubdomains` | `boolean` | `false` | Follow subdomains of the main domain. | | `delay` | `number` (s) | — | Delay between scrapes. Setting this forces concurrency to 1. | #### Sitemap and deduplication | Parameter | Type | Default | Description | | - | - | - | - | | `sitemap` | `string` | `"include"` | `"include"`: use sitemap + link discovery. `"skip"`: ignore sitemap. `"only"`: crawl only sitemap URLs. | | `deduplicateSimilarURLs` | `boolean` | `true` | Normalize URL variants (`www.`, `https`, trailing slashes, `index.html`) as duplicates. | | `ignoreQueryParameters` | `boolean` | `false` | Strip query strings before deduplication (e.g. `/page?a=1` and `/page?a=2` become one URL). | #### Scrape options for crawl | Parameter | Type | Default | Description | | - | - | - | - | | `scrapeOptions` | `object` | `{ formats: ["markdown"] }` | Per-page scrape config. Accepts all [scrape options](#scrape-options) above. | ### Crawl example ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/crawl \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://docs.firecrawl.dev", "includePaths": ["^/blog/.*$", "^/docs/.*$"], "excludePaths": ["^/admin/.*$", "^/private/.*$"], "maxDiscoveryDepth": 2, "limit": 1000 }' ``` ## Mapping website links The `/v2/map` endpoint identifies URLs related to a given website. ```bash cURL theme={null} curl -X POST https://api.firecrawl.dev/v2/map \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer fc-YOUR-API-KEY' \ -d '{ "url": "https://docs.firecrawl.dev" }' ``` ### Map options | Parameter | Type | Default | Description | | - | - | - | - | | `search` | `string` | — | Filter links by text match. | | `limit` | `integer` | `100` | Max links to return. | | `sitemap` | `string` | `"include"` | `"include"`, `"skip"`, or `"only"`. | | `includeSubdomains` | `boolean` | `true` | Include subdomains. | Here is the API Reference for it: [Map Endpoint Documentation](https://docs.firecrawl.dev/api-reference/endpoint/map) ## Whitelisting Firecrawl ### Allowing Firecrawl to scrape your website * **robots.txt**: Firecrawl reads `robots.txt` before it crawls a site, and applies the rules written for the user-agent token `FirecrawlAgent` as well as the rules written for `*`. Use `FirecrawlAgent` to allow or disallow Firecrawl in `robots.txt`. * **User agent**: `FirecrawlAgent` is the token Firecrawl matches against when it applies `robots.txt` rules. Do not assume it is also the `User-Agent` header on the fetch itself. Allow-list Firecrawl with a `robots.txt` directive rather than with a firewall or WAF rule that matches a `User-Agent` string. * **IP addresses**: Firecrawl does not use a fixed set of outbound IPs. ### Allowing your application to call the Firecrawl API If your firewall blocks outbound requests from your application to external services, you need to whitelist Firecrawl's API server IP address so your application can reach the Firecrawl API (`api.firecrawl.dev`): * **IP Address**: `35.245.250.27` Add this IP to your firewall's outbound allowlist so your backend can send scrape, crawl, map, and agent requests to Firecrawl. # Billing Source: https://docs.firecrawl.dev/billing How Firecrawl billing, credits, and plans work ## Overview Firecrawl billing is **credit-based**. Every API call that you make consumes credits. The number of credits depends on the endpoint and the options that you use. Your plan gives you a monthly credit allotment. Pay-as-you-go adds more credits when the allotment runs out. For current plan pricing, visit the [Firecrawl pricing page](https://www.firecrawl.dev/pricing). > [!NOTE] > > All Firecrawl invoices are billed in **US Dollars (USD)**, regardless of your billing address or payment method. ## Credits Credits are the unit of usage in Firecrawl. Each plan includes a monthly credit allotment that resets at the start of each billing cycle. Different API endpoints consume different amounts of credits. ### Credit costs per endpoint | Endpoint | Credit Cost | Notes | | - | - | - | | **Scrape** | 1 credit / page | Convert a single URL into clean markdown, HTML, or structured data. Additional credits apply when using scrape options (see below). | | **Crawl** | 1 credit / page | Scrape an entire website by following links from a starting URL. The same per-page scrape option costs apply to each page crawled. | | **Map** | 1 credit / call | Discover all URLs on a website without scraping their content. | | **Search** | 2 credits / 10 results | Search the web and optionally scrape the results. Rounded up per 10 results (e.g., 11 results = 4 credits). Additional per-page scrape costs apply to each result that is scraped. See [here](/features/search#zero-data-retention-zdr) for enterprise ZDR search pricing. | | **Interact** | 2–7 credits / browser minute | Interactive browser sandbox session, billed per browser minute with a one-minute minimum. Sessions that use a `prompt` bill at 7 credits / browser minute; sessions without a prompt (Playwright `code` only) bill at 2 credits / browser minute. [Zero Data Retention](/features/interact#zero-data-retention-zdr) sessions add 2 credits / browser minute (9 with a prompt, 4 code-only). | | **Agent** | Dynamic | Autonomous web research agent. 5 daily runs free; usage-based pricing beyond that. See [here](/features/agent#zdr-pricing) for ZDR agent pricing. | ### Additional credit costs for scrape options Certain scrape options add credits on top of the base cost per page: | Option | Additional Cost | Description | | - | - | - | | PDF parsing | +1 credit / PDF page | Extract content from PDF documents | | JSON format (LLM extraction) | +4 credits / page | Use an LLM to extract structured JSON data from the page | | Prompt injection check | +4 credits / page | Opt-in `checkPromptInjection` scrape option, available with any format (see [Prompt injection detection](/features/llm-extract#prompt-injection-detection)). If the scrape fails after the check has run, 5 credits are billed. See [When credits are charged](#when-credits-are-charged). | | Zero Data Retention (ZDR) | +1 credit / page | Ensures no data is persisted beyond the request (see [Scrape ZDR](/features/scrape#zero-data-retention-zdr)) | | `question` or `query` format | +4 credits / page | LLM-generated answer to a question about the page (see [Scrape](/features/scrape)) | | `highlights` format | +4 credits / page | LLM-selected relevant passages from the page (see [Scrape](/features/scrape)) | | `audio` format | +4 credits / page | Transcribe audio found on the page (see [Scrape](/features/scrape)) | | `video` format | +4 credits / page | Transcribe video found on the page (see [Scrape](/features/scrape)) | | PII redaction (`redactPII`) | +4 credits / page | Redact personal data from the returned markdown. Each additional PDF page adds another +4 on top of its +1 parsing cost (see [PII redaction](/features/pii-redaction)) | | `lockdown` (cache hit) | +4 credits / page | Serve from cache only, never fetching the target. A cache miss returns no document and bills 1 credit (see [Lockdown](/features/lockdown) and the table below) | | Enhanced proxy (`proxy: "enhanced"` or `auto` escalation) | +0 | Billed at the same 1 credit as a basic request. An escalated retry is not charged separately (see [Enhanced Mode](/features/enhanced-mode)) | These modifiers stack. For example, scraping a page with both JSON format and Zero Data Retention costs **1 + 4 + 1 = 6 credits** per page, and JSON format with PII redaction costs **1 + 4 + 4 = 9 credits**. These same modifiers apply to the Crawl and Search endpoints since they use scrape internally for each page. Requests to `x.com` and other X/Twitter URLs use the Grok API and have separate pricing. See [X (x.com) billing](#x-xcom-billing) at the bottom of this page. LinkedIn profile and company pages are answered by third-party data providers and priced by them. See [Provider-routed URLs](#provider-routed-urls) at the bottom of this page. ### When credits are charged What decides the charge is whether Firecrawl returned a document, not whether the target site returned a successful HTTP status code. * **Firecrawl returned a document: 1 credit per page**, plus any of the option costs above. This includes pages that come back with an error status such as 403 Forbidden or 404 Not Found. The target responded, Firecrawl captured that response, and you get it back as a document. Check the `metadata.statusCode` field in the response to spot these cases and stop retrying URLs that are consistently blocked. * **Firecrawl returned no document: 0 credits.** A scrape that fails outright, for example because the site never responded or every rendering attempt failed, is not charged. A few cases still charge when no document comes back. They cover work that Firecrawl already performed on your behalf before the scrape ended. | Case | What is charged | | - | - | | Threat protection scan | 2 credits per scanned URL, including a scrape that the scan itself blocked | | Prompt injection check | 5 credits, once the `checkPromptInjection` guard has run | | v1 scrape `agent` option | Usage-based, for the navigation the agent already did | | `lockdown` cache miss | 1 credit | | Monitor check | A page that errors during a check is charged the base 1 credit per page | | Interact and Browser sessions | Charged per browser minute of session time, with a one-minute minimum, whether or not the session got what you wanted | For **batch scrape** and **crawl** jobs, credits are billed asynchronously as each page completes processing, not when the job is submitted. This means there can be a delay between submitting a job and seeing the full credit cost reflected on your account. If a batch contains many URLs or pages are queued during high-traffic periods, credits may continue to appear minutes or hours after submission. Polling or checking batch status does not consume credits. > [!NOTE] > > **Crawl pre-flight credit check:** When you pass an explicit `limit`, Firecrawl checks whether the request is allowed for that amount. If it is allowed, including through overage, the limit is not reduced to your remaining balance. If it is denied, Firecrawl attempts to lower the limit to the remaining balance and checks the adjusted request again. A 402 is returned if no positive limit can be used or the adjusted request is denied, including by a per-API-key spend limit. > > Omitting `limit` does not require **10,000** credits up front. The initial credit check uses **1 credit**; it does not reserve credits for the full crawl. If the request is allowed, the default limit of **10,000** is not reduced to fit the balance, although other crawl options can change the effective limit. Pages are billed as they are processed. Pass an explicit `limit` when you want to set a maximum number of pages (e.g., `limit: 100`). ### Tracking your usage You can monitor your credit usage in two ways: * **Dashboard**: View your current and historical usage at [firecrawl.dev/app](https://www.firecrawl.dev/app) * **API**: Use the [Credit Usage](/api-reference/endpoint/credit-usage) and [Credit Usage Historical](/api-reference/endpoint/credit-usage-historical) endpoints to programmatically check your usage > [!NOTE] > > We are actively working on improvements to make credit usage easier to understand. Stay tuned for updates. ## Plans Subscription plans bill monthly or yearly. Paid self-serve plans can also use pay-as-you-go. Pay-as-you-go adds credits when your plan allotment runs out. See [Pay-as-you-go](#pay-as-you-go). ### Paid plans | Plan | Monthly Credits | Concurrent Browsers | | - | - | -: | | **Hobby** | 5,000 | 5 | | **Standard** | 100,000 | 25 | | **Growth** | 500,000 | 50 | | **Scale** | 1,000,000 | 100 | > [!NOTE] > > For needs beyond Scale, Firecrawl offers **Enterprise** plans with custom credits, dedicated support, SLAs, bulk discounts, zero-data retention, and SSO. Visit the [Enterprise page](https://www.firecrawl.dev/enterprise) for details. All paid plans are available with **monthly** or **yearly** billing. Yearly billing offers a discount compared to paying month-to-month. For current pricing on each plan, visit the [pricing page](https://www.firecrawl.dev/pricing). ### Free plan New accounts start on the free plan. It costs nothing and does not need a card. Here is what it includes: * **1,000 credits per month.** The allotment resets at the start of each monthly cycle. Unused free credits do not roll over. * **2 concurrent browsers**, with the same 50,000 maximum queued jobs as the paid self-serve plans. * **Rate limits of 10 requests per minute** on `/scrape`, `/map` and `/search`, and **2 requests per minute** on `/crawl`, `/agent` and `/interact`. Batch scrape shares the scrape limit and extract shares the agent limit. See [Rate Limits](/rate-limits) for the full table. * **Every core product endpoint.** Scrape, crawl, map, search, extract, batch scrape, interact and monitor all work on the free plan. Some team administration and enterprise features are enabled per team rather than by plan, including threat protection, SIEM logging and zero data retention search. Credit costs are the same on every plan, so the [credit costs per endpoint](#credit-costs-per-endpoint) table above applies unchanged. The free plan does not include pay-as-you-go, so requests return an HTTP 402 once the monthly allotment runs out. See [Running Out of Credits](#running-out-of-credits). It also does not include credit rollover, a Data Processing Agreement, or Slack support. Support is community support. Scrape, Search and Interact also work with no API key at all, through the keyless free tier, which is capped per IP address per day rather than by a credit allotment. See [Keyless (no API key)](/rate-limits#keyless-no-api-key). ### Billing cycle * **Monthly plans**: Credits reset on your monthly renewal date * **Yearly plans**: You are billed annually, but credits still reset each month on your virtual monthly renewal date * **Unused plan credits do not roll over by default**: your monthly allotment resets each month. **Annual Scale plans roll unused plan credits over 1 month**, and **annual Enterprise plans roll them over 2 months**. ### Concurrent browsers Concurrent browsers represent how many web pages Firecrawl can process for you simultaneously. Your plan determines this limit. If you exceed it, additional jobs wait in a queue until a slot opens. See [Rate Limits](/rate-limits) for full details on concurrency and API rate limits. ## Pay-as-you-go Pay-as-you-go keeps your requests running when your plan credits run out. It adds credits to your account automatically. You buy credits in increments of 5 USD. When your credit balance reaches zero, we add one increment to your account and charge your card on file. Pay-as-you-go needs a paid self-serve plan. You cannot use pay-as-you-go on the free plan. You can also buy credits yourself at any time. Use **Load more credits** in your [billing settings](https://www.firecrawl.dev/app/settings?tab=billing). Enter a multiple of 5 USD, and pay with your card on file. This works whether pay-as-you-go is on or off. ### Credits in an increment The credits in an increment depend on your plan. Pay-as-you-go and manual purchases use the same rate. | Plan | Credits per 5 USD | | - | - | | **Hobby** | 1,000 | | **Standard** | 2,000 | | **Growth** | 2,500 | | **Scale** | 5,000 | ### Set the monthly pay-as-you-go limit Set your **Monthly pay-as-you-go limit** in either of these two places: * In your [billing settings](https://www.firecrawl.dev/app/settings?tab=billing), on the **Billing** tab. * On the [pricing page](https://www.firecrawl.dev/pricing), when you pick a plan. ### How the limit caps your monthly spend Your limit is the most that pay-as-you-go can spend in one month. It rounds down to whole 5 USD increments. For example, a limit of 25 USD allows five increments each month. A limit of 22 USD allows four increments, because 22 USD rounds down to 20 USD. Credits that you buy manually do not count toward this limit. ## API key spend limits A spend limit caps the credits that one API key can use. The cap applies to that key alone, whatever your team balance is. Use it to stop one key from spending the whole team balance. You can also rename any API key. This includes the key named **Default** that Firecrawl creates with your team. The key owner or a team admin can do both. Open the [API keys page](https://www.firecrawl.dev/app/api-keys) in the dashboard, then open the menu on the key. * Click **Rename** to change the **Key Name**. * Click **Set limit** to open **Set a spend limit**. Enter a **Credit limit**, pick **per day**, **per week** or **per month**, then click **Save**. A limit is a number of credits, not an amount of money. A limit of `1` means one credit per period. Click **Remove limit** to clear the limit. ## Upgrading and Downgrading * **Upgrades** take effect immediately. You are charged the full new-plan price today (no proration), and your billing cycle resets. Your next renewal is one month or one year from the upgrade date. Any unused credits from your previous plan carry over, and your new credit allotment and concurrency limits apply right away. * **Downgrades** are scheduled to take effect at your next renewal date. You keep your current plan's credits and limits until then, and unused time on your current plan is not credited or refunded. You can undo a scheduled downgrade from your [billing settings](https://www.firecrawl.dev/app/settings?tab=billing) any time before the effective date. ### Switching between monthly and yearly billing * **Monthly → Yearly** at the same or higher credit tier is treated as an immediate upgrade. * **Yearly → Monthly** is treated as an immediate upgrade only if you move to a strictly higher credit tier. ## Running Out of Credits What happens when your balance reaches zero depends on your plan and on your pay-as-you-go setting. | Plan | Pay-as-you-go | At zero balance | | - | - | - | | **Free** | Not available | Requests that consume credits return **HTTP 402 (Payment Required)** until the monthly reset. | | **Hobby, Standard, Growth, Scale** | Off (limit set to `0`) | Requests that consume credits return **HTTP 402**. Your card is not charged. | | **Hobby, Standard, Growth, Scale** | On | Firecrawl charges your card for one 5 USD increment and adds its credits. Your requests continue. This repeats each time the balance reaches zero, until your **monthly pay-as-you-go limit** is reached. Leave the limit blank for no cap. | While pay-as-you-go is on, requests are not cut off at exactly zero. Your plan carries an overage allowance so that a burst of requests keeps running while a top-up settles: **1,500 credits on Hobby, 30,000 on Standard, 150,000 on Growth, and 600,000 on Scale**. Once the monthly limit is reached and the allowance is used, requests return **HTTP 402** until the next billing cycle. The allowance is headroom, not extra credits you keep. Manually purchased credits (**Load more credits**) are spent before any of this applies and do not count toward the monthly pay-as-you-go limit. To resume usage after a hard stop, you can: 1. Set a **Monthly pay-as-you-go limit** to buy credits automatically. See [Pay-as-you-go](#pay-as-you-go). 2. Upgrade to a higher plan manually 3. Wait for your credits to reset at the next billing cycle ## Coupons Firecrawl supports two types of coupons: * **Subscription coupons** apply a discount to your plan subscription (e.g. a percentage off your monthly or yearly price). These can **only** be applied during the Stripe checkout flow when you first subscribe to a paid plan or change plans. You cannot apply a subscription coupon after checkout has completed. * **Credit coupons** add bonus credits to your account. These can be redeemed from the **Billing** section of your dashboard at [firecrawl.dev/app/billing](https://www.firecrawl.dev/app/billing). Look for the coupon input field on the billing page to apply your code. Bonus credits from credit coupons are separate from your plan's monthly allotment and persist even if you upgrade or downgrade your plan. ## FAQs > [accordion] Do unused credits roll over to the next month? > > **Plan credits** do not roll over by default: your monthly allotment resets each month. **Annual Scale plans roll unused plan credits over 1 month**, and **annual Enterprise plans roll them over 2 months**. > [accordion] Do pay-as-you-go credits expire? > > Credits you buy stay on your account until you use them. They expire if you cancel your subscription, so use them before you leave. The expiry lands at the end of your current billing period, and a renewal does not touch them. > [accordion] How does the monthly pay-as-you-go limit work? > > Your limit caps what pay-as-you-go spends each month. It rounds down to whole 5 USD increments. A limit of 25 USD allows five increments each month. Leave your limit blank, and pay-as-you-go has no monthly limit. Set your limit to `0`, and pay-as-you-go turns off. > [accordion] How do I know how many credits I have left? > > Check the dashboard at [firecrawl.dev/app](https://www.firecrawl.dev/app), or call the [Credit Usage API endpoint](/api-reference/endpoint/credit-usage) programmatically. > [accordion] Where do I apply a coupon code? > > It depends on the coupon type. Apply a **credit coupon** in the Billing section of your dashboard. You can apply a **subscription coupon** (a discount on your plan price) only at the Stripe checkout page, when you subscribe or change plans. > [accordion] I need more concurrency or a custom plan. Who do I contact? > > Reach out to [help@firecrawl.dev](mailto:help@firecrawl.dev), or visit the [Enterprise page](https://www.firecrawl.dev/enterprise) to learn more about custom plans. > [accordion] What currency are invoices billed in? > > All Firecrawl invoices are billed in **US Dollars (USD)**, regardless of your billing address or payment method. > [accordion] How do I manage my subscription, change plans, or cancel? > > Go to your [billing settings](https://www.firecrawl.dev/app/settings?tab=billing). Team admins can manage everything there. Click **Manage Subscription** to open the billing portal and update your payment method, billing address, company name, or VAT number. > > To change plans, click **Change Plan** and pick a new tier. Upgrades take effect immediately. Downgrades take effect at the end of your current billing period, and you can undo one until then. See [Upgrading and Downgrading](#upgrading-and-downgrading). > > To cancel, click **Cancel Subscription**. Your plan stays active until the end of your current billing period, and you can resume it before then. When your plan ends, the credits you bought expire with it, so use them before you leave. > [accordion] How do I add a VAT number, company name, or billing address to my invoices? > > Go to your [billing settings](https://www.firecrawl.dev/app/settings?tab=billing), click **Manage Subscriptions**, and update your billing address, company name, and VAT number in the Stripe portal. Future invoices will automatically include the updated details. > > To regenerate a past paid invoice with the new information: > > 1. Update your billing details in the Stripe portal first (see above). > 2. Open the **Invoice history** tab in the Stripe portal and download the PDF for the invoice you want; Stripe re-renders it against your current billing info. > 3. If an invoice doesn't pick up the updated details, email [help@firecrawl.dev](mailto:help@firecrawl.dev) with the invoice numbers and we'll regenerate them for you. ## Provider-routed URLs A scrape of a LinkedIn person profile or company page is answered by the third-party data providers your org chose in the enrichment settings (see [Provider-routed URLs](/features/scrape#provider-routed-urls)). It costs the credits those providers charged, in place of the 1-credit scrape base. | Provider | Capability | Person profile (`linkedin.com/in/…`) | Company page (`linkedin.com/company/…`) | | - | - | - | - | | Apollo | `people/match`, `companies/enrich` | 30 credits | 30 credits | | Data Legion (base) | `people/enrich-base-no-contact`, `companies/enrich-base` | 10 credits | 10 credits | | Data Legion (premium) | `people/enrich-premium-no-contact`, `companies/enrich-premium` | 40 credits | 40 credits | | FullEnrich | `people/lookup`, `companies/lookup` | 5 credits | 5 credits | * Providers run in your saved order, and the first one that returns a record ends the run. The request costs the sum of what each step charged, shown in `metadata.provider.creditsCost` and broken down in `metadata.provider.steps`. * A provider with no record for the URL charges nothing, so a request normally costs the price of the provider that matched. For example, with Data Legion (base) first and Apollo second, a profile Data Legion has costs 10 credits, and one only Apollo has costs 30. When no provider has a record, the request fails with `THIRD_PARTY_DATA_NOT_FOUND` and costs nothing. * Format surcharges still apply. A `json` scrape of a profile that Apollo serves costs 30 + 4 = 34 credits. `zeroDataRetention`, `lockdown` and `redactPII` can't be used on these URLs, so their surcharges never apply. * Every request is charged. `x-request-id` doesn't replay URL scrapes. The enrichment settings in the dashboard show the capability and price each provider uses for your team. ## X (x.com) billing Firecrawl uses the official **Grok API** from [xAI](https://x.ai/) to provide AI-powered summarization, structured extraction, and real-time access to public X content. Requests to `x.com`, `twitter.com`, and `mobile.twitter.com` profile and post URLs are handled through Grok's authorized internal tools (`x_search`, thread fetch, and web search restricted to x.com) rather than traditional web scraping. ### Credit costs | Component | Credit Cost | Description | | - | - | - | | **Base cost** | 1 credit / request | Standard scrape request processing | | **Grok X Query** | +29 credits / request | Grok API usage (tokens + tool calls) for X content | For example, processing a typical post or thread request costs **30 credits** (`1` base + `29` Grok X Query) and returns Grok-generated structured data, thread context, and summaries. If JSON format (LLM extraction) is also enabled, the total is **34 credits** per request. This method complies with X's published interfaces via xAI's partnership and provides higher-quality, reasoned output instead of raw page scraping. > [!NOTE] > > **Capabilities differ from standard scraping.** Grok returns AI-processed results, which may include summaries, key metrics, thread context, and more. For raw structured data at scale, use the [official X Enterprise API](https://developer.x.com/). # Building an AI Research Assistant with Firecrawl and AI SDK Source: https://docs.firecrawl.dev/developer-guides/cookbooks/ai-research-assistant-cookbook Build a complete AI-powered research assistant with web scraping and search capabilities Build a complete AI-powered research assistant that can scrape websites and search the web to answer questions. The assistant automatically decides when to use web scraping or search tools to gather information, then provides comprehensive answers based on collected data.

>
> > > {messages.map((message) => ( >
> {message.role === "assistant" && > message.parts.filter((part) => part.type === "source-url") > .length > 0 && ( > > count={ > message.parts.filter( > (part) => part.type === "source-url" > ).length > } > /> > {message.parts > .filter((part) => part.type === "source-url") > .map((part, i) => ( > > key={`${message.id}-${i}`} > href={part.url} > title={part.url} > /> > > ))} > > )} > {message.parts.map((part, i) => { > switch (part.type) { > case "text": > return ( > > > > {part.text} > > > {message.role === "assistant" && > i === messages.length - 1 && ( > > onClick={() => regenerate()} > label="Retry" > > > > > onClick={() => > navigator.clipboard.writeText(part.text) > } > label="Copy" > > > > > > )} > > ); > case "reasoning": > return ( > key={`${message.id}-${i}`} > className="w-full" > isStreaming={ > status === "streaming" && > i === message.parts.length - 1 && > message.id === messages.at(-1)?.id > } > > > > {part.text} > > ); > default: { > if (part.type.startsWith("tool-")) { > const toolPart = part as ToolUIPart; > return ( > key={`${message.id}-${i}`} > defaultOpen={toolPart.state === "output-available"} > > > type={toolPart.type} > state={toolPart.state} > /> > > > output={toolPart.output} > errorText={toolPart.errorText} > /> > > > ); > } > return null; > } > } > })} >
> ))} > {status === "submitted" && } >
> >
> > onSubmit={handleSubmit} > className="mt-4" > globalDrop > multiple > > > > > {(attachment) => } > > > > onChange={(e) => setInput(e.target.value)} > value={input} > /> > > > > > > > > > > variant={webSearch ? "default" : "ghost"} > onClick={() => setWebSearch(!webSearch)} > > > > Search > > onValueChange={(value) => { > setModel(value); > }} > value={model} > > > > > > > {models.map((model) => ( > key={model.value} > value={model.value} > > > {model.name} > > ))} > > > > > > >
>
> ); > }; > > export default ChatBotDemo; > ``` ### Understanding the Frontend The frontend uses AI Elements components to provide a complete chat interface: **Key Features:** * **Conversation Display**: The `Conversation` component automatically handles message scrolling and display * **Message Rendering**: Each message part is rendered based on its type (text, reasoning, tool calls) * **Tool Visualization**: Tool calls are displayed with collapsible sections showing inputs and outputs * **Interactive Controls**: Users can toggle web search, select models, and attach files * **Message Actions**: Copy and retry actions for assistant messages **4. Add Markdown Rendering Support** To ensure the markdown from the LLM is correctly rendered, add the following import to your `app/globals.css` file: ```css theme={null} @source "../node_modules/streamdown/dist/index.js"; ``` This imports the necessary styles for rendering markdown content in the message responses. **5. Build the Basic API Route** Create the chat API endpoint at `app/api/chat/route.ts`. This route will handle incoming messages and stream responses from the AI. ```typescript theme={null} import { streamText, UIMessage, convertToModelMessages } from "ai"; import { createOpenAI } from "@ai-sdk/openai"; const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY!, }); // Allow streaming responses up to 5 minutes export const maxDuration = 300; export async function POST(req: Request) { const { messages, model, webSearch, }: { messages: UIMessage[]; model: string; webSearch: boolean; } = await req.json(); const result = streamText({ model: openai(model), messages: convertToModelMessages(messages), system: "You are a helpful assistant that can answer questions and help with tasks.", }); // send sources and reasoning back to the client return result.toUIMessageStreamResponse({ sendSources: true, sendReasoning: true, }); } ``` This basic route: * Receives messages from the frontend * Uses the OpenAI model selected by the user * Streams responses back to the client * Doesn't include tools yet - we'll add those next **6. Configure Environment Variables** Create a `.env.local` file in your project root: ```bash theme={null} touch .env.local ``` Add your OpenAI API key: ```env theme={null} OPENAI_API_KEY=sk-your-openai-api-key ``` The `OPENAI_API_KEY` is required for the AI model to function. **7. Test the Basic Chat** Now you can test the AI SDK chatbot without Firecrawl integration. Start the development server: ```bash theme={null} npm run dev ``` Open [localhost:3000](http://localhost:3000) in your browser and test the basic chat functionality. The assistant should respond to messages, but won't have web scraping or search capabilities yet.