Crawlbase Web MCP Server

Created By

crawlbase4 months ago

Crawlbase Web MCP Server (Model Context Protocol) connects AI agents and large language models (LLMs) with real-time web data. Built on Crawlbase’s proven web scraping and crawling infrastructure, it handles JavaScript rendering, anti-bot protection, and web data extraction at scale — powering Claude, Cursor, and Windsurf. It delivers a reliable, production-ready pipeline of live, structured data for AI workflows and intelligent applications.

# web-scraping

# web-scraping-mcp

Overview Content Tools Comments

Content

Why Crawlbase MCP?

Large Language Models (LLMs) like Claude and ChatGPT are powerful, but disconnected from the live web. They can’t fetch real-time data, leading to outdated answers, hallucinations, and limited agent workflows.

Crawlbase MCP Server fixes this by bridging AI tools with fresh, structured web data — directly inside Claude Desktop, Cursor IDE, and Windsurf. Built on Crawlbase’s proven scraping infrastructure (trusted by 70,000+ developers), it delivers:

Real-time web scraping
JavaScript rendering for dynamic apps
Proxy rotation & anti-bot evasion
Structured outputs (HTML, Markdown, Screenshots)

Setup & Integration

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Cursor IDE

Add to .cursor-settings.json:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Windsurf IDE

Add to MCP settings:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

🔑 Get your free tokens at Crawlbase ↗.

Usage

Once configured, use these commands inside Claude, Cursor, or Windsurf:

crawl → Fetch raw HTML
crawl_markdown → Extract clean Markdown
crawl_screenshot → Capture screenshots

Example prompts:

“Crawl Hacker News and return top stories in markdown.”
“Take a screenshot of TechCrunch homepage.”
“Fetch Tesla investor relations page as HTML.”

Use Cases

Market research → Pull live data from competitors, news, and reports
E-commerce monitoring → Track products, reviews, and prices in real time
News & finance feeds → Keep AI agents up-to-date with live events
Autonomous AI agents → Give them vision to act on fresh web data

Resources & Next Steps

Looking to supercharge your AI agents with live web data? Get started here:

Server Config

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "@crawlbase/mcp@latest"
      ],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Recommend Servers

TraeBuild with Free GPT-4.1 & Claude 3.7. Fully MCP-Ready.

Howtocook Mcp基于Anduin2017 / HowToCook （程序员在家做饭指南）的mcp server，帮你推荐菜谱、规划膳食，解决“今天吃什么“的世纪难题； Based on Anduin2017/HowToCook (Programmer's Guide to Cooking at Home), MCP Server helps you recommend recipes, plan meals, and solve the century old problem of "what to eat today"

WindsurfThe new purpose-built IDE to harness magic

Zhipu Web SearchZhipu Web Search MCP Server is a search engine specifically designed for large models. It integrates four search engines, allowing users to flexibly compare and switch between them. Building upon the web crawling and ranking capabilities of traditional search engines, it enhances intent recognition capabilities, returning results more suitable for large model processing (such as webpage titles, URLs, summaries, site names, site icons, etc.). This helps AI applications achieve "dynamic knowledge acquisition" and "precise scenario adaptation" capabilities.

AiimagemultistyleA Model Context Protocol (MCP) server for image generation and manipulation using fal.ai's Stable Diffusion model.

Baidu Map百度地图核心API现已全面兼容MCP协议，是国内首家兼容MCP协议的地图服务商。

DeepChatYour AI Partner on Desktop

Tavily Mcp

Amap Maps高德地图官方 MCP Server

Jina AI MCP ToolsA Model Context Protocol (MCP) server that integrates with Jina AI Search Foundation APIs.

ChatWiseThe second fastest AI chatbot™