Sponsored by Deepsite.site, Most Affordable AI Code Agent Powered by DeepSeek.

Crawler For Llm

Created By

KongHaYao6 months ago

高等级的免费爬虫MCP, 完全 Markdown 化支持，支持微信公众号等一大批国内网站。A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.

# Web Crawler

# Markdown Converter

Overview Content Tools Comments

Overview

Crawler for LLM with MCP Support

A universal web crawler for LLM frameworks with Model Context Protocol (MCP) integration

A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.

{
    "mcpServers": {
        "langgraph-crawler": {
            "command": "npx",
            "args": ["-y", "@langgraph-js/crawler-mcp@latest"]
        }
    }
}

Supported Websites

Note: This list is continuously expanding. Feel free to contribute by adding support for more websites.

Documentation & Development

Chinese Platforms

License

Apache-2.0

Server Config

{
  "mcpServers": {
    "langgraph-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "@langgraph-js/crawler-mcp@latest"
      ]
    }
  }
}

Recommend Servers

TraeBuild with Free GPT-4.1 & Claude 3.7. Fully MCP-Ready.

Visual Studio Code - Open Source ("Code - OSS")Visual Studio Code

MCP AdvisorMCP Advisor & Installation - Use the right MCP server for your needs

DeepChatYour AI Partner on Desktop

Zhipu Web SearchZhipu Web Search MCP Server is a search engine specifically designed for large models. It integrates four search engines, allowing users to flexibly compare and switch between them. Building upon the web crawling and ranking capabilities of traditional search engines, it enhances intent recognition capabilities, returning results more suitable for large model processing (such as webpage titles, URLs, summaries, site names, site icons, etc.). This helps AI applications achieve "dynamic knowledge acquisition" and "precise scenario adaptation" capabilities.

BlenderBlenderMCP connects Blender to Claude AI through the Model Context Protocol (MCP), allowing Claude to directly interact with and control Blender. This integration enables prompt assisted 3D modeling, scene creation, and manipulation.

Amap Maps高德地图官方 MCP Server

WindsurfThe new purpose-built IDE to harness magic

Tavily Mcp

EdgeOne Pages MCPAn MCP service designed for deploying HTML content to EdgeOne Pages and obtaining an accessible public URL.

Baidu Map百度地图核心API现已全面兼容MCP协议，是国内首家兼容MCP协议的地图服务商。