XCrawl quickly turns search results and website content into LLM-friendly formats.

Welcome to XCrawl

XCrawl is an API service that accepts URLs and keywords, crawls and searches, and delivers results in multiple LLM-friendly formats including Markdown. Our API supports both single-page scraping and full-site crawling. You can also search by keyword and crawl relevant pages.

• Scrape: Fetch a single URL and return Markdown / HTML / Links / Summary / Screenshot / JSON (structured extraction)
• Crawl: Crawl a site in bulk (async)
• Map: List all URLs within a site
• Search: Search by keyword with location/language support (advanced Google SERP parameters)

How to use

You can start in the dashboard Playground, then move to the docs for production usage.

Recommended reading:

• Authentication
• Scrape API
• Crawl API
• Search API

API Key

To use the XCrawl API, you need an API key. After signing up, get it from the dashboard.

Feature overview

• Scrape: Fetch a URL and return LLM-friendly formats (Markdown, HTML, screenshots, links, summaries, extracted JSON)
• Crawl: Crawl all URLs under a site and return LLM-friendly content
• Map: List all URLs for a site to quickly understand structure
• Search: Search by keyword with region and language controls

Scrape

Use the scrape endpoint to fetch a single URL and return the requested formats.

curl -s -X POST 'https://run.xcrawl.com/v1/scrape'   -H 'Authorization: Bearer $XCRAWL_API_KEY'  -H 'Content-Type: application/json'   -d '{
    "url": "https://example.com",
    "output": {
      "formats": ["markdown"]
    }
  }'

Example response:

{
  "scrape_id": "01KKE88ETDN4RE9J7EPC5HR89B",
  "endpoint": "scrape",
  "version": "dca0d4b3bff035e4",
  "status": "completed",
  "url": "https://example.com",
  "data": {
    "markdown": "# Example Domain\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n[Learn more](https://iana.org/domains/example)",
    "metadata": {
      "content_type": "text/html",
      "final_url": "https://example.com/",
      "status_code": 200,
      "title": "Example Domain"
    },
    "traffic_bytes": 1410,
    "credits_used": 1
  },
  "started_at": "2026-03-11T10:49:39Z",
  "ended_at": "2026-03-11T10:49:44Z",
  "total_credits_used": 1
}

Crawl

The crawl endpoint discovers and fetches all URLs under a site. Provide a start URL and optional crawl rules.

curl -s -X POST 'https://run.xcrawl.com/v1/crawl'   -H 'Authorization: Bearer $XCRAWL_API_KEY'  -H 'Content-Type: application/json'   -d '{
    "url": "https://docs.xcrawl.com/doc/",
    "crawler": {
      "limit": 1,
      "max_depth": 1
    },
    "output": {
      "formats": ["markdown"]
    }
  }'

Example response:

{
  "crawl_id": "01KKE8BNNVQH9PCYEEKJGXKE07",
  "endpoint": "crawl",
  "version": "dca0d4b3bff035e4",
  "status": "crawling"
}

Check crawl status and results

When you create a crawl task, you receive a crawl_id. Use it to query status and results.

curl -s -X GET 'https://run.xcrawl.com/v1/crawl/01KKE8BNNVQH9PCYEEKJGXKE07' \
  -H 'Authorization: Bearer $XCRAWL_API_KEY'

Example response:

{
  "crawl_id": "01KKE8BNNVQH9PCYEEKJGXKE07",
  "endpoint": "crawl",
  "version": "dca0d4b3bff035e4",
  "status": "completed",
  "completed": 1,
  "total": 1,
  "url": "https://docs.xcrawl.com/doc/",
  "data": [
    {
      "markdown": "[ Skip to content ](https://docs.xcrawl.com/doc/developer-guides/proxies/#VPContent)\n# Proxy Setup\nXCrawl supports proxy configuration to choose an exit region or reuse sticky sessions.\n...",
      "metadata": {
        "statusCode": 200,
        "title": "Proxy Setup",
        "url": "https://docs.xcrawl.com/doc/developer-guides/proxies/"
      },
      "traffic_bytes": 1129,
      "credits_used": 1
    }
  ],
  "started_at": "2026-03-11T10:51:24Z",
  "ended_at": "2026-03-11T10:51:37Z",
  "total_credits_used": 1
}

Search

The search endpoint lets you fetch results by keyword with region and language controls, including advanced Google SERP options.

curl -s -X POST 'https://run.xcrawl.com/v1/search'   -H 'Authorization: Bearer $XCRAWL_API_KEY'  -H 'Content-Type: application/json'   -d '{
    "query": "site:docs.xcrawl.com XCrawl API",
    "location": "US",
    "language": "en",
    "limit": 2
  }'

Example response:

{
  "search_id": "01KKE8BNMEKRHJB9GEWXPYQ8E1",
  "endpoint": "search",
  "version": "dca0d4b3bff035e4",
  "status": "completed",
  "query": "site:docs.xcrawl.com XCrawl API",
  "data": {
    "credits_used": 2,
    "data": [
      {
        "position": 1,
        "title": null,
        "url": "https://docs.xcrawl.com/"
      },
      {
        "position": 2,
        "title": null,
        "url": "https://docs.xcrawl.com/doc/developer-guides/authentication/"
      }
    ],
    "status": "success"
  },
  "started_at": "",
  "ended_at": "",
  "total_credits_used": 2
}

AI extraction

XCrawl's Scrape endpoint can extract JSON without providing a schema. Just describe what you want in natural language.

curl -s -X POST 'https://run.xcrawl.com/v1/scrape'   -H 'Authorization: Bearer $XCRAWL_API_KEY'  -H 'Content-Type: application/json'   -d '{
    "url": "https://example.com",
    "output": {
      "formats": ["json"]
    },
    "json": {
      "prompt": "Extract the page title and main CTA link."
    }
  }'

Example response:

{
  "scrape_id": "01KKE8JTFD20YP1KXPVJHGCS28",
  "endpoint": "scrape",
  "version": "dca0d4b3bff035e4",
  "status": "completed",
  "url": "https://example.com",
  "data": {
    "json": {
      "description": "This domain is for use in documentation examples without needing permission. Avoid use in operations.",
      "domain": {
        "name": "example",
        "permissions_required": false,
        "purpose": "documentation examples"
      },
      "links": [
        {
          "text": "Learn more",
          "url": "https://iana.org/domains/example"
        }
      ],
      "title": "Example Domain"
    },
    "credits_used": 5,
    "credits_detail": {
      "base_cost": 1,
      "traffic_cost": 0,
      "json_extract_cost": 4
    }
  },
  "started_at": "2026-03-11T10:55:19Z",
  "ended_at": "2026-03-11T10:55:26Z",
  "total_credits_used": 5
}

Next pageScrape