Site Search Opener MCP

Research your own site and many competitor sites from inside your AI app. Ask a question in plain words, and get a table of real pages: URL, title, description, last modified date and HTTP status.

It reads what websites publish for search engines: their sitemaps and RSS or Atom feeds. It checks robots.txt first, goes slowly, and says who it is. When a site has nothing to read, you get Google search links to open in your own browser instead.

Works with Claude (Desktop, Code and claude.ai), ChatGPT, Gemini CLI, Gemini Enterprise, OpenAI Codex, Cursor, VS Code with GitHub Copilot, Windsurf and any other app that supports MCP.

This is the AI companion to the free Site Search Opener web app by bestSEO.tips. Project files move between the two.

• Install guide

• All recipes

• Host your own copy on Cloudflare

What you can ask

Which of my competitors have pages about maintenance backlog? Show me a table.

How is competitor.com's content organised? Which folders are biggest?

What did my competitors publish in the last 90 days?

Find case studies on rival-a.com and rival-b.com for the food industry.

Build a project file from the competitors we talked about before, and check they all work.

Which of my own pages haven't been updated in a year?

Is anyone on Reddit talking about our product category? Give me links.

Related MCP server: Webpage MCP Server

How it works

1. You ask in plain words. The AI picks a recipe, such as content-gap or case-studies, and your domains.

2. The server reads each site's own files. robots.txt first, then the sitemaps it lists (or the usual places, like /sitemap.xml), then the site's RSS or Atom feed. Pages that robots.txt asks bots to avoid are left out.

3. You get a table. Matching pages with URL, title, description, date and HTTP status, plus folder counts or pages per month for some recipes.

4. No sitemap? You get links. If a site has no usable sitemap, blocks bots, or nothing matched, the server makes Google site: search links for that site. They open in your browser, the same as typing the search yourself.

What it never does:

• It never searches Google or reads Google results. The links are for you to click.

• It never tries to get past a CAPTCHA, a bot check or a robots.txt rule.

• It never sends your data to bestseo.tips or anyone else. The local version runs on your computer. The hosted version runs in your own Cloudflare account.

Site owners can opt out at any time by adding this to their robots.txt:

User-agent: SiteSearchOpenerMCP
Disallow: /

Install

The quickest local setup: install Node.js 18.17 or newer, download this repository, and point your app at:

node <folder>/plugin/server/site-search-opener.mjs

Tools

Recipes

34 recipes for seven teams. A few examples:

The full list, with what each one reads, is in docs/RECIPES.md.

Slash commands

In Claude Code with the plugin, type /site-search-opener: and pick one:

Other apps show the same commands as MCP prompts, usually in a prompt or attachment menu.

Projects and the web app

A project is a named list of your own domains and competitor domains, in the same JSON format the web app uses:

{
  "app": "sitesearch-opener",
  "version": 2,
  "projects": [
    { "name": "HR software", "own": ["linkedin.com"], "competitors": ["indeed.com", "xing.com"] }
  ]
}

• Ask the AI to build one. It can collect domains you mentioned before, check each one returns HTTP 200, and follow redirects.

• Import it in the web app: open https://bestseo.tips/tools/site-search-opener/, go to Settings, then Export/Import Projects, then Import.

• Export from the web app and hand the file to the AI, so it uses the same lists.

Limits

The local version waits at least one second between requests to the same site, longer if the site's robots.txt asks for a Crawl-delay.

Privacy and safety

• The server only fetches public http and https addresses. It refuses local and private network addresses, and redirects that lead to them.

• Requests carry the user agent SiteSearchOpenerMCP/1.0 (+https://github.com/albertalbs/site-search-opener-mcp).

• Downloads are capped in size and time.

• Text from websites is passed to the AI as data, and the AI is told not to follow instructions inside it.

See SECURITY.md for more.

Questions

Why doesn't it just search Google for me? Google's terms do not allow automated searches, and it blocks them. Sitemaps are published so machines can read them, so that is what this tool reads. For Google results, it gives you the links.

Does it use bestseo.tips servers? No. The local version runs on your computer. The hosted version runs in your Cloudflare account. bestseo.tips only hosts the web app.

Why does a site show links instead of a table? Either the site has no sitemap or feed the tool can find, robots.txt asks bots to stay away, or nothing matched your filters. The links still let you check it yourself.

Why do I need a project file? You don't, but it saves typing the same 20 competitors every time, and the web app reads the same file.

Development

npm install
npm test                 # unit tests and MCP protocol tests
npm run build            # rebuilds plugin/server/site-search-opener.mjs and copies the skill
npm run docs:recipes     # rebuilds docs/RECIPES.md from src/core/recipes.js
npx wrangler dev         # runs the Cloudflare version locally

Layout:

src/core/      shared logic: domains, queries, recipes, robots.txt, feeds, inventory, projects
src/server.js  the MCP server, shared by local and hosted versions
src/stdio.js   local version (adds save-file)
worker/        Cloudflare Workers version
skill/         the Agent Skill
plugin/        the Claude Code plugin (commands, skill copy, bundled server)
docs/          install, hosting and recipe guides
test/          tests, with a fake internet so they never touch real sites

Contributions are welcome, especially new recipes. See CONTRIBUTING.md.

License and credits

Apache License 2.0. See LICENSE and NOTICE.

If you share or build on this project, keep the NOTICE file with it, as the license asks. A link back is appreciated.

Created by Albert Adaikkalam, bestSEO.tips. LinkedIn: https://www.linkedin.com/in/albertalbs/

Third-party packages in the bundled server are listed in THIRD_PARTY_NOTICES.md.