Map

Map quickly retrieves a list of URLs within a site.

List site URLs before crawling for selective capture
Quickly understand site structure
Estimate the number of accessible pages

List all URLs under a domain with XCrawl

/map endpoint

Usage

curl -s -X POST 'https://run.xcrawl.com/v1/map'   -H 'Authorization: Bearer $XCRAWL_API_KEY'  -H 'Content-Type: application/json'   -d '{
    "url": "https://docs.xcrawl.com/doc/",
    "limit": 5
  }'

Response example

{
  "map_id": "01KKE88ET3XC7HMPE4CNQNCGSW",
  "endpoint": "map",
  "version": "dca0d4b3bff035e4",
  "status": "completed",
  "url": "https://docs.xcrawl.com/doc/",
  "data": {
    "links": [
      "https://docs.xcrawl.com/",
      "https://docs.xcrawl.com/doc/",
      "https://docs.xcrawl.com/zh/",
      "https://docs.xcrawl.com/doc/introduction/",
      "https://docs.xcrawl.com/zh/doc/"
    ],
    "total_links": 5,
    "credits_used": 1
  },
  "started_at": "2026-03-11T10:49:39Z",
  "ended_at": "2026-03-11T10:49:40Z",
  "total_credits_used": 1
}

Filter results

Use filter to return only URLs matching a regex. For example, return only /case/ paths:

curl -s -X POST 'https://run.xcrawl.com/v1/map'   -H 'Authorization: Bearer $XCRAWL_API_KEY'  -H 'Content-Type: application/json'   -d '{
    "url": "https://xcrawl.com",
    "filter": "/case/.*"
    }'