mcp-server-webcrawl

Created By

pragmar8 months ago

Bridge the gap between your web crawler and AI language models using Model Context Protocol (MCP). With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Support for WARC, wget, InterroBot, Katana, and SiteOne crawlers is available out of the gate. The server includes a full-text search interface with boolean support, resource filtering by type, HTTP status, and more.

# web-crawler

# web-scraping

Overview Content Tools Comments

Overview

what is mcp-server-webcrawl?

mcp-server-webcrawl is an open-source server that bridges the gap between web crawlers and AI language models using the Model Context Protocol (MCP). It allows AI clients to filter and analyze web content, extracting insights either under user direction or autonomously.

how to use mcp-server-webcrawl?

To use mcp-server-webcrawl, install it via pip with the command: pip install mcp-server-webcrawl. You can then run the server using the command: mcp-server-webcrawl --crawler wget --datasrc /path/to/wget/archives/.

key features of mcp-server-webcrawl?

Compatibility with Claude Desktop
Full-text search interface with boolean support
Resource filtering by type and HTTP status
Support for various crawlers including wget, WARC, and more
Ability to augment your LLM knowledge base
ChatGPT support is coming soon

use cases of mcp-server-webcrawl?

Analyzing web content for research purposes
Extracting insights from large datasets collected by web crawlers
Enhancing AI language models with real-time web data

FAQ from mcp-server-webcrawl?

Is mcp-server-webcrawl free to use?

Yes! mcp-server-webcrawl is free and open-source.

What are the system requirements?

It requires Claude Desktop and Python version 3.10 or higher.

Which crawlers are supported?

It supports wget, WARC, InterroBot, Katana, and SiteOne crawlers.

Server Config

{
  "mcpServers": {
    "webcrawl": {
      "command": "mcp-server-webcrawl",
      "args": [
        "--crawler",
        "wget",
        "--datasrc",
        "/path/to/wget/archives/"
      ]
    }
  }
}

Recommend Servers

TraeBuild with Free GPT-4.1 & Claude 3.7. Fully MCP-Ready.

WindsurfThe new purpose-built IDE to harness magic

Baidu Map百度地图核心API现已全面兼容MCP协议，是国内首家兼容MCP协议的地图服务商。

Visual Studio Code - Open Source ("Code - OSS")Visual Studio Code

ChatWiseThe second fastest AI chatbot™

BlenderBlenderMCP connects Blender to Claude AI through the Model Context Protocol (MCP), allowing Claude to directly interact with and control Blender. This integration enables prompt assisted 3D modeling, scene creation, and manipulation.

TimeA Model Context Protocol server that provides time and timezone conversion capabilities. This server enables LLMs to get current time information and perform timezone conversions using IANA timezone names, with automatic system timezone detection.

AiimagemultistyleA Model Context Protocol (MCP) server for image generation and manipulation using fal.ai's Stable Diffusion model.

Amap Maps高德地图官方 MCP Server

Tavily Mcp

Howtocook Mcp基于Anduin2017 / HowToCook （程序员在家做饭指南）的mcp server，帮你推荐菜谱、规划膳食，解决“今天吃什么“的世纪难题； Based on Anduin2017/HowToCook (Programmer's Guide to Cooking at Home), MCP Server helps you recommend recipes, plan meals, and solve the century old problem of "what to eat today"