Web Content MCP

This MCP server is called web-content-mcp and provides two tools:

• web-search: Searches Google to get the top-most results, and scrapes data from those pages by visiting them first
• url: Directly visits the provided URL(s) and scrapes content without searching Google

Features

• Fast and efficient web content retrieval
• Support for both search-based and direct URL access
• Compatible with Claude Desktop, VSCode, and any agent that supports MCP servers

Prerequisites

• Node.js and pnpm/npm installed
• TypeScript support
• A .env file with required API keys (see Configuration section)

Installation

1. Clone this repository

git clone https://github.com/yourusername/web-content-mcp.git
cd web-content-mcp

2. Install dependencies

pnpm install

3. Configure your .env file (see Configuration section)

Configuration

Create a .env file in the root directory with the following environment variables:

GOOGLE_API_KEY=your_google_api_key
GOOGLE_SEARCH_ENGINE_ID=your_google_search_engine_id

Setup in Claude Desktop

Add the following configuration to your Claude Desktop configuration file:

{
  "mcpServers": {
    "web-content-mcp": {
      "command": "pnpx",
      "args": [
        "tsx",
        "--env-file=/path/to/your/.env",
        "/path/to/your/web-content-mcp/src/index.ts"
      ]
    }
  }
}

For Windows users, use something like:

{
  "mcpServers": {
    "web-content-mcp": {
      "command": "pnpx",
      "args": [
        "tsx",
        "--env-file=C:\\Users\\username\\Projects\\web-content-mcp\\.env",
        "C:\\Users\\username\\Projects\\web-content-mcp\\src\\index.ts"
      ]
    }
  }
}

Setup in VSCode

Add the following to your VSCode's settings.json (User Settings preferred):

{
  "mcp": {
    "servers": {
      "web-content-mcp": {
        "type": "stdio",
        "command": "pnpx",
        "args": [
          "tsx",
          "--env-file=/path/to/your/.env",
          "/path/to/your/web-content-mcp/src/index.ts"
        ]
      }
    }
  }
}

For Windows users, make sure to escape backslashes:

{
  "mcp": {
    "servers": {
      "web-content-mcp": {
        "type": "stdio",
        "command": "pnpx",
        "args": [
          "tsx",
          "--env-file=C:\\\\Users\\\\username\\\\Projects\\\\web-content-mcp\\\\.env",
          "C:\\\\Users\\\\username\\\\Projects\\\\web-content-mcp\\\\src\\\\index.ts"
        ]
      }
    }
  }
}

Usage

Once configured, the MCP server can be accessed in Claude prompts:

Using web-search

Search for information about [search query]

Using direct URL access

Visit and summarize information from [URL]

Troubleshooting

• Connection errors: Make sure the server is running and the path to your files is correct
• API errors: Verify that your API keys in the .env file are valid
• Path issues: Double-check the paths in your configuration files

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.