Sail MCP Server for Spark SQL
Sail
The mission of Sail is to unify stream processing, batch processing, and compute-intensive (AI) workloads. Currently, Sail features a drop-in replacement for Spark SQL and the Spark DataFrame API in both single-host and distributed settings.
✨News✨: Please check out our MCP server that brings data analytics in Spark to both LLM agents and humans!
Installation
Sail is available as a Python package on PyPI. You can install it using pip.
pip install "pysail[spark]"
Alternatively, you can install Sail from source for better performance for your hardware architecture. You can follow the Installation guide for more information.
Getting Started
Starting the Sail Server
Option 1: Command Line Interface You can start the local Sail server using the sail command.
sail spark server --port 50051
Option 2: Python API You can start the local Sail server using the Python API.
from pysail.spark import SparkConnectServer
server = SparkConnectServer(port=50051)
server.start(background=False)
Option 3: Kubernetes You can deploy Sail on Kubernetes and run Sail in cluster mode for distributed processing. Please refer to the Kubernetes Deployment Guide for instructions on building the Docker image and writing the Kubernetes manifest YAML file.
kubectl apply -f sail.yaml
kubectl -n sail port-forward service/sail-spark-server 50051:50051
Connecting to the Sail Server
Once you have a running Sail server, you can connect to it in PySpark. No changes are needed in your PySpark code!
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:50051").getOrCreate()
spark.sql("SELECT 1 + 1").show()
Please refer to the Getting Started guide for further details.
Documentation
The documentation of the latest Sail version can be found here.
Further Reading
- Supercharge Spark: Quadruple Speed, Cut Costs by 94% - This post presents detailed benchmark results comparing Sail with Spark.
- Sail 0.2 and the Future of Distributed Processing - This post discusses the Sail distributed processing architecture.
Contributing
Contributions are more than welcome!
Please submit GitHub issues for bug reports and feature requests. You are also welcome to ask questions in GitHub discussions.
Feel free to create a pull request if you would like to make a code change. You can refer to the development guide to get started.
Support
LakeSail offers flexible enterprise support options for Sail. Please contact us to learn more.
Server Config
{
"mcpServers": {
"sail": {
"command": "sail",
"args": [
"spark",
"mcp-server",
"--transport",
"stdio"
]
}
}
}