Sponsored by Deepsite.site, Most Affordable AI Code Agent Powered by DeepSeek.

Crawler For Llm

Created By

KongHaYao6 months ago

高等级的免费爬虫MCP, 完全 Markdown 化支持，支持微信公众号等一大批国内网站。A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.

# Web Crawler

# Markdown Converter

Overview Content Tools Comments

Content

Crawler for LLM with MCP Support

A universal web crawler for LLM frameworks with Model Context Protocol (MCP) integration

A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.

{
    "mcpServers": {
        "langgraph-crawler": {
            "command": "npx",
            "args": ["-y", "@langgraph-js/crawler-mcp@latest"]
        }
    }
}

Supported Websites

Note: This list is continuously expanding. Feel free to contribute by adding support for more websites.

Documentation & Development

Chinese Platforms

License

Apache-2.0

Server Config

{
  "mcpServers": {
    "langgraph-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "@langgraph-js/crawler-mcp@latest"
      ]
    }
  }
}

Recommend Servers

TraeBuild with Free GPT-4.1 & Claude 3.7. Fully MCP-Ready.

Zhipu Web SearchZhipu Web Search MCP Server is a search engine specifically designed for large models. It integrates four search engines, allowing users to flexibly compare and switch between them. Building upon the web crawling and ranking capabilities of traditional search engines, it enhances intent recognition capabilities, returning results more suitable for large model processing (such as webpage titles, URLs, summaries, site names, site icons, etc.). This helps AI applications achieve "dynamic knowledge acquisition" and "precise scenario adaptation" capabilities.

ChatWiseThe second fastest AI chatbot™

Serper MCP ServerA Serper MCP Server

Amap Maps高德地图官方 MCP Server

Baidu Map百度地图核心API现已全面兼容MCP协议，是国内首家兼容MCP协议的地图服务商。

Tavily Mcp

EdgeOne Pages MCPAn MCP service designed for deploying HTML content to EdgeOne Pages and obtaining an accessible public URL.

TimeA Model Context Protocol server that provides time and timezone conversion capabilities. This server enables LLMs to get current time information and perform timezone conversions using IANA timezone names, with automatic system timezone detection.

DeepChatYour AI Partner on Desktop

CursorThe AI Code Editor