Web Scraper MCP

Created By

navin40786 months ago

Scrape websites and let them talk to your LLM

Overview

What is MCP Web Scraper?

MCP Web Scraper is a lightweight and efficient web scraping server that allows users to scrape websites and interact with their data using the Model Context Protocol (MCP).

How to use MCP Web Scraper?

To use MCP Web Scraper, you can either automate the setup by cloning the repository and running the setup script, or manually set it up by creating a virtual environment and installing the required dependencies.

Key features of MCP Web Scraper?

Text, link, image, and table data extraction with CSS selectors.
Comprehensive metadata extraction including Open Graph and Twitter Cards.
Integration with Claude Desktop for seamless operation.
Configurable result limits and error handling.

Use cases of MCP Web Scraper?

Extracting text content from various websites.
Gathering headlines and metadata for news articles.
Scraping images and tables for data analysis.

FAQ from MCP Web Scraper?

Can MCP Web Scraper handle all types of websites?

Yes, it can scrape a wide variety of websites as long as they allow it in their robots.txt file.

Is there a limit to the number of results I can scrape?

Yes, you can configure the maximum number of results to prevent overload.

What dependencies does MCP Web Scraper require?

It requires libraries like requests, beautifulsoup4, and lxml for web scraping.

Server Config

{
  "mcpServers": {
    "web-scraper": {
      "command": "/full/path/to/your/venv/bin/python",
      "args": [
        "/full/path/to/your/app_mcp.py"
      ]
    }
  }
}

Recommend Servers

TraeBuild with Free GPT-4.1 & Claude 3.7. Fully MCP-Ready.

AiimagemultistyleA Model Context Protocol (MCP) server for image generation and manipulation using fal.ai's Stable Diffusion model.

TimeA Model Context Protocol server that provides time and timezone conversion capabilities. This server enables LLMs to get current time information and perform timezone conversions using IANA timezone names, with automatic system timezone detection.

Howtocook Mcp基于Anduin2017 / HowToCook （程序员在家做饭指南）的mcp server，帮你推荐菜谱、规划膳食，解决“今天吃什么“的世纪难题； Based on Anduin2017/HowToCook (Programmer's Guide to Cooking at Home), MCP Server helps you recommend recipes, plan meals, and solve the century old problem of "what to eat today"

Jina AI MCP ToolsA Model Context Protocol (MCP) server that integrates with Jina AI Search Foundation APIs.

EdgeOne Pages MCPAn MCP service designed for deploying HTML content to EdgeOne Pages and obtaining an accessible public URL.

CursorThe AI Code Editor

Tavily Mcp

Amap Maps高德地图官方 MCP Server

MiniMax MCPOfficial MiniMax Model Context Protocol (MCP) server that enables interaction with powerful Text to Speech, image generation and video generation APIs.

Zhipu Web SearchZhipu Web Search MCP Server is a search engine specifically designed for large models. It integrates four search engines, allowing users to flexibly compare and switch between them. Building upon the web crawling and ranking capabilities of traditional search engines, it enhances intent recognition capabilities, returning results more suitable for large model processing (such as webpage titles, URLs, summaries, site names, site icons, etc.). This helps AI applications achieve "dynamic knowledge acquisition" and "precise scenario adaptation" capabilities.