Claude Code + XCrawl MCP Setup Guide

This guide covers integrating the XCrawl MCP Server into Claude Code, giving Claude Code built-in web scraping, search, URL discovery, and bulk crawling capabilities.

Prerequisites

Get an XCrawl API Key

Sign up at dash.xcrawl.com
Receive 1,000 free credits upon registration
Get your API Key at dash.xcrawl.com/api-key

Requirements

Claude Code installed
For local npx method: Node.js >= 18
For cloud URL method: no additional dependencies

Method 1: Cloud-Hosted URL

This is the recommended setup. It has no local dependencies and works out of the box.

claude mcp add --transport http xcrawl https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp

With scope control:

# Global, available in all projects
claude mcp add --scope user --transport http xcrawl https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp

# Project-level, written to .mcp.json and trackable by git
claude mcp add --scope project --transport http xcrawl https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp

Method 2: Local npx

Use this for offline or intranet environments, or when you need custom control.

claude mcp add xcrawl -e XCRAWL_API_KEY={YOUR_API_KEY} -- npx -y xcrawl-mcp

Method 3: Project-Level Config File

Create .mcp.json in your project root:

{
  "mcpServers": {
    "xcrawl": {
      "type": "http",
      "url": "https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp"
    }
  }
}

Or with npx. Use environment variables to avoid exposing keys:

{
  "mcpServers": {
    "xcrawl": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "xcrawl-mcp"],
      "env": {
        "XCRAWL_API_KEY": "${XCRAWL_API_KEY}"
      }
    }
  }
}

Note

.mcp.json supports ${VAR} and ${VAR:-default} syntax for environment variable expansion, avoiding hard-coded API keys in git.

Method 4: Claude Desktop

For using XCrawl MCP in Claude Desktop instead of the CLI:

Config file paths:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

HTTP method:

{
  "mcpServers": {
    "xcrawl": {
      "url": "https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp"
    }
  }
}

Note

If your Claude Desktop version does not support HTTP, use the npx method below.

npx method:

{
  "mcpServers": {
    "xcrawl": {
      "command": "npx",
      "args": ["-y", "xcrawl-mcp"],
      "env": {
        "XCRAWL_API_KEY": "{YOUR_API_KEY}"
      }
    }
  }
}

Verify Configuration

# List configured MCP servers
claude mcp list

# View xcrawl details
claude mcp get xcrawl

Inside a Claude Code session, type /mcp to check connection status and available tools.

Management Commands

claude mcp list             # List all MCP servers
claude mcp get xcrawl       # Show xcrawl config details
claude mcp remove xcrawl    # Remove xcrawl

Available Tools

Once configured, Claude Code gains access to 7 XCrawl tools:

Tool	Description
`xcrawl_scrape`	Single-page scraping with markdown/html/json/screenshot output
`xcrawl_search`	Google SERP search with location/language/advanced parameters
`xcrawl_map`	Site URL discovery with regex filtering and subdomain support
`xcrawl_crawl`	Bulk crawling with depth and path control
`xcrawl_key_status`	Check API Key validity and credit balance
`xcrawl_check_status`	Query async scrape task status
`xcrawl_check_crawl_status`	Query async crawl task status

Usage Examples

# Single-page scrape
Use xcrawl_scrape to fetch https://example.com in sync mode and return markdown and links.

# Search
Use xcrawl_search to search for "Claude Code MCP" in US English and return the top 10 results.

# URL discovery
Use xcrawl_map to list /doc/ URLs under https://docs.xcrawl.com with a limit of 2000.

# Bulk crawl
Use xcrawl_crawl to start a crawl from https://docs.xcrawl.com/doc/ with max depth 2 and limit 100, then poll until it completes.

Troubleshooting

Issue	Solution
Server connection timeout	Set `MCP_TIMEOUT=30000` to extend the timeout
`spawn npx ENOENT`	Install Node.js LTS and ensure `npx` is in PATH
HTTP connection failure	Your client version may not support HTTP. Switch to npx
Tools not showing up	Run `/mcp` in session to check server status
Output truncated	Set `MAX_MCP_OUTPUT_TOKENS=50000`
401 Unauthorized	Invalid API Key. Reissue it at dash.xcrawl.com
Insufficient credits	Call `xcrawl_key_status` to check, then upgrade your plan
Health check	`curl https://mcp.xcrawl.com/health` should return `OK`

SERP

Engines

Data

APIs