Local Grounding MCP

Privacy-first visual grounding fallback with local inference and DOM-visual hybrid routing for MCP browser agents.

Experimental. Currently tested on macOS Apple Silicon with Python 3.11, Apple MPS, and UGround-V1-2B.

Why

Browser agents should use reliable DOM and accessibility references whenever possible. When structured references are unavailable or unreliable, Local Grounding MCP provides a local visual fallback for canvas interfaces, unlabeled controls, image-based elements, and other difficult UI targets.

DOM / accessibility first
          |
          | no reliable structured reference
          v
Local visual grounding fallback

Related MCP server: pic-id-mcp

Architecture

MCP client A adapter ----\
                          > Shared localhost service -> one local model
MCP client B adapter ----/

The shared service provides:

• one model process shared across MCP clients;

• lazy model loading on the first model request;

• serialized model loading and inference;

• automatic service startup;

• configurable idle shutdown;

• screenshot deletion after visual tensors are created;

• local token and latency metrics;

• localhost-only networking by default.

Current platform support

Tested:

• macOS on Apple Silicon;

• Python 3.11;

• Apple MPS;

• UGround-V1-2B;

• Claude Desktop and Antigravity as MCP clients.

Not yet tested: Intel Mac, Linux/CUDA, Windows, CPU-only inference, and other grounding models.

Requirements

• macOS on Apple Silicon;

• Python 3.11 available as python3.11;

• Git;

• sufficient local storage and unified memory;

• UGround-V1-2B model weights downloaded separately.

Model weights are not included. Obtain UGround-V1-2B from its publisher and review its current terms:

https://huggingface.co/osunlp/UGround-V1-2B

Default location:

~/Models/UGround-V1-2B

Alternative location:

export GROUNDING_MODEL_PATH="/absolute/path/to/UGround-V1-2B"

Installation

git clone https://github.com/archie-ib-dev/local-grounding-mcp.git
cd local-grounding-mcp
./scripts/install.sh

The installer creates an isolated environment at ~/.local/share/local-grounding-mcp/venv and command links under ~/.local/bin. Add ~/.local/bin to PATH if necessary.

Verify:

local-grounding-doctor

MCP configuration

The installer automatically runs:

local-grounding-config

It creates ready-to-paste snippets at:

~/.config/local-grounding-mcp/claude-desktop.json
~/.config/local-grounding-mcp/antigravity.json

The snippets use absolute paths and do not overwrite existing client configuration. Merge only the generated local-grounding entry into the existing mcpServers object. Static examples are also available in examples/.

Commands

local-grounding-doctor
local-grounding-status
local-grounding-stop
local-grounding-metrics
local-grounding-config

Lower-level commands:

local-grounding-mcp
local-grounding-service

An MCP adapter normally starts the service automatically. Status checks do not intentionally load the model. Stopping the process releases the model with that process.

MCP tools

• grounding_health: return status without intentionally loading the model;

• load_grounding_model: explicitly load the shared model;

• locate_target: locate a described target in a screenshot;

• cleanup_grounding_screenshots: remove orphan screenshots by age.

Visual grounding is a fallback. Clients should inspect DOM and accessibility information first.

Configuration

Common environment variables:

GROUNDING_CLIENT
GROUNDING_HOST
GROUNDING_PORT
GROUNDING_IDLE_SECONDS
GROUNDING_MODEL_PATH
GROUNDING_SCREENSHOT_DIR
GROUNDING_CACHE_DIR
GROUNDING_LOG_DIR
GROUNDING_SERVICE_SCRIPT
GROUNDING_SERVICE_LOG
GROUNDING_START_LOCK
GROUNDING_CONFIG_DIR

The default host is 127.0.0.1. Do not expose the service on 0.0.0.0 without authentication and authorization.

Privacy and telemetry

Inference is local. Screenshots may briefly exist on disk. With delete_after_read=true, the consumed screenshot is deleted after processing inputs are created and before inference completes.

Metrics may include timestamps, operation names, client labels, process IDs, image dimensions, token counts, timings, deletion status, and success status. The current metrics design is not intended to store screenshot contents, page text, URLs, cookies, credentials, browser-extension tokens, or target descriptions. Review logs and metrics before sharing.

Baseline measurement

One validated clean-environment run on Apple Silicon, MPS, and FP32 produced:

Image size:              753 x 620
Visual tokens:           1,044
Input tokens:            1,121
Generated tokens:        11
Model load:              5.304 seconds
Preprocessing:           0.026 seconds
Inference:               8.347 seconds
Cold-start total:        13.685 seconds
Screenshot deleted:      true

Results vary with hardware, dependencies, image dimensions, model state, and workload.

Uninstall

./scripts/uninstall.sh

The uninstaller removes the environment and command links. It preserves model weights, logs, metrics, generated snippets, and client configuration.

Security

See SECURITY.md. Keep the service on localhost, do not commit secrets or client configuration, and review automation before allowing access to sensitive applications.

License

Local Grounding MCP is licensed under the GNU Affero General Public License version 3 only (AGPL-3.0-only). Commercial use is allowed under the AGPLv3, subject to its terms.

The license is intended to keep modifications to the covered runtime available under the same open-source terms, including where AGPL network-interaction obligations apply. Organizations requiring different terms for proprietary use may contact the copyright holder to discuss alternative licensing. No alternative commercial license or commercial terms are granted through this repository.

Third-party software and separately downloaded models retain their own licenses. See THIRD_PARTY_NOTICES.md.

Contributions

Issues, documentation, tests, and code contributions are welcome. Unless a separate written agreement applies, accepted contributions are expected to be provided under AGPL-3.0-only. A Contributor License Agreement may be required before accepting work intended for an alternatively licensed edition. See CONTRIBUTING.md.

Status

Experimental and under active development. Platform support, installation behavior, security boundaries, and dependency compatibility may change before a stable release.