This guide covers integrating the XCrawl MCP Server into OpenAI Codex, giving Codex built-in web scraping, search, URL discovery, and bulk crawling capabilities.

Prerequisites

Get an XCrawl API Key

1. Sign up at dash.xcrawl.com
2. Receive 1,000 free credits upon registration
3. Get your API Key at dash.xcrawl.com/api-key

Requirements

• Codex CLI or Codex VS Code extension installed
• For stdio method: Node.js >= 18
• For HTTP method: Codex v0.44.0 or later

Method 1: CLI Command

stdio Method

codex mcp add xcrawl --env XCRAWL_API_KEY={YOUR_API_KEY} -- npx -y xcrawl-mcp

Note

Options (--env, --url, etc.) go before the -- separator. Everything after -- is the command and arguments passed to the MCP server.

HTTP Method

Requires Codex v0.44.0 or later.

codex mcp add xcrawl --url https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp

Method 2: Direct config.toml Edit

Config file location: ~/.codex/config.toml

stdio Method

[mcp_servers.xcrawl]
command = "npx"
args = ["-y", "xcrawl-mcp"]
env = { "XCRAWL_API_KEY" = "{YOUR_API_KEY}" }
startup_timeout_sec = 30
tool_timeout_sec = 120

HTTP Method

Requires Codex v0.44.0 or later.

[mcp_servers.xcrawl]
url = "https://mcp.xcrawl.com/{YOUR_API_KEY}/mcp"
startup_timeout_sec = 30
tool_timeout_sec = 120

Codex VS Code Extension

The Codex VS Code extension automatically reads ~/.codex/config.toml. No additional configuration is needed to use configured XCrawl MCP tools within the extension.

Note

Each MCP server is initialized at Codex startup. Configuring too many servers can slow down the extension launch.

Environment Variable Configuration

Option A: Explicit in config.toml

[mcp_servers.xcrawl]
command = "npx"
args = ["-y", "xcrawl-mcp"]
env = { "XCRAWL_API_KEY" = "{YOUR_API_KEY}" }

Option B: Shell environment variable

Set the variable before launch:

export XCRAWL_API_KEY={YOUR_API_KEY}
codex

With this approach, the env field can be omitted from config.toml:

[mcp_servers.xcrawl]
command = "npx"
args = ["-y", "xcrawl-mcp"]

Verify Configuration

# List all MCP servers
codex mcp list

# View xcrawl details
codex mcp get xcrawl

Management Commands

codex mcp list             # List all MCP servers
codex mcp get xcrawl       # Show xcrawl config details
codex mcp remove xcrawl    # Remove xcrawl

Available Tools

Once configured, Codex gains access to 7 XCrawl tools:

Tool	Description
xcrawl_scrape	Single-page scraping with markdown/html/json/screenshot output
xcrawl_search	Google SERP search with location/language/advanced parameters
xcrawl_map	Site URL discovery with regex filtering and subdomain support
xcrawl_crawl	Bulk crawling with depth and path control
xcrawl_key_status	Check API Key validity and credit balance
xcrawl_check_status	Query async scrape task status
xcrawl_check_crawl_status	Query async crawl task status

Usage Examples

# Single-page scrape
Use xcrawl_scrape to fetch https://example.com in sync mode and return markdown and links.

# Search
Use xcrawl_search to search for "Codex MCP integration" in US English and return the top 10 results.

# URL discovery
Use xcrawl_map to list /doc/ URLs under https://docs.xcrawl.com with a limit of 2000.

# Bulk crawl
Use xcrawl_crawl to start a crawl from https://docs.xcrawl.com/doc/ with max depth 2 and limit 100, then poll until it completes.

Troubleshooting

Issue	Solution
spawn npx ENOENT	Install Node.js LTS, ensure npx is in PATH, restart terminal
MCP server not appearing	Check config.toml syntax. The section must be [mcp_servers.xcrawl]
HTTP method not working	Verify Codex version >= 0.44.0
Slow startup	Reduce unnecessary MCP server entries in config.toml
Environment variable not taking effect	Use env = {} to explicitly specify it in config.toml
Tools unavailable after extension launch	Restart VS Code or reload the window
401 Unauthorized	Invalid API Key. Reissue it at dash.xcrawl.com
Insufficient credits	Call xcrawl_key_status to check, then upgrade your plan
Health check	curl https://mcp.xcrawl.com/health should return OK
Codex silently hides MCP errors	Validate that the MCP server works in Claude Code first, then debug the Codex config

Previous pageOpenClaw

Next pageClaude Code MCP